We used to run something called a game day. Now, this is basically a simulation of how an actual match is going. We used to generate not just traffic. We are simulating the entire operating protocol. We'll say that, okay, the match is going to start at 7 p.m. Let's say, okay. Now, every system is supposed to scale beforehand. There's a timeline to it. There is a checklist to it and so on. No matter whether the teams are ready or not, we will start the live stream and we'll start sending traffic. Wow, really? You did that? They don't know what the traffic is going to come, what kind of a pattern is going to come and anything because that's the exact set of our production. You don't know what is coming your way. Ashutosh Agarwal was a software architect and principal engineer at the largest streaming service in India, Geo Cinema, now known as Disney Plus Hotstar.
In the middle of 2023, the system he architected set a world record at the time with 32 million parallel connected live streams during the finale of the Indian Premier League cricket tournament. But how did they do it? In today's episode, we cover how live streaming works behind the scenes and how large-scale live streaming systems are architected. The trade-offs in system design and how server-side load, stream leniency, and stream smoothness are all connected. The importance of capacity planning and drills, including what the game day drill meant at GeoCinema, and many more details. If you're interested about large-scale systems or live streaming, this episode is for you. If you enjoy the show, please subscribe to the podcast on any podcast platform and on YouTube. Welcome to the podcast. Thank you. Thank you, Greg. Can you help us imagine what it was like going back to the day this world record was set with 32 million, roughly that many Concord connections at the time?
What was it like? You know, like on the day, you knew this big match was coming up. There was a system that you did architect, you know, preparing it for scale. But then, you know, like you're getting a larger than ever spike. What happened? Were you in the office? Was it at home? Was it stressful? Was it chill? Sure.
Most of the world records that you have set in past are generally set on the finale of the event, right? And by finale, you've probably settled into the event. These events are not like one-off or one-day event like how it happens on other platforms. Indian Premier League is like a 70-day event. So it's a 70-day continuous madness, right? And every day is more than 15 to 20 million viewers watching and depending on the game, depending on the players who are coming in. So it's not like that. To be aware, the finale is relatively more calmer than the opening week. The opening week is where the most of the madness is there. By finale, you understand the traffic pattern better. You understand what your requirements are. Your protocols are much more set in depth.
In the initial part of the game, it is all chill till you're operating under the numbers which you planned for. The moment you start to see it getting into the yellow zone is when things become interesting because then you have to kind of get away from the automotive, like a car driving on running the platform to a manual mode, which requires you to be looking at all the metrics, looking at every issue which is coming in, ensuring that you're trying it on a... on a very short period of time. Otherwise, what will happen is that they will become noise. So you have to also ensure that the ongoing issues as a clean board. And in terms of setup, we typically prefer to be in the office into the same location because if there is an incident, it is much easier to get into a room with all the key people and be able to take calls and make decisions. We have done the setup remotely as well. We had a strict protocol on how we operate remotely to ensure that there's not much of a madness. It's very easy to call the Zoom call to go crazy. The Zoom call on which all of us are connected is over 200 people with multiple partners across the globe present over there to support the event. It's just not our team.
my team. It's like multiple partners or partners who are present on the call who are helping us scale through the event. This episode was brought to you by WorkOS. If you're building a SaaS app, at some point your customers will start asking for enterprise features like SAM authentication, SKIM provisioning, and fine-grained authorization. That's where WorkOS comes in, making it fast and painless to add enterprise features to your app. Their APIs are easy to understand and you can ship quickly and get back to building other features.
WorkOS also provides a free user management solution called AuthKit for up to 1 million monthly active users. It's a drop-in replacement for Auth0 and comes standard with useful features like domain verification, role-based access control, bot protection, and MFA. It's powered by Radix components, which means zero compromises in design. You get limited customizations as well as modular templates designed for quick integrations. Today, hundreds of fast-growing startups are powered by WorkOS, including ones you probably know, like Cursor, Vercel, and Perplexity. Check it out at WorkOS.com to learn more. That is WorkOS.com. This episode was brought to you by CodeRabbit, the AI code review platform transforming how engineering teams ship faster without sacrificing code quality.
CodeRabbit acts as your AI co-pilot, providing instant code review comments and potential impacts of every pull request. Beyond just flagging issues, CodeRabbit provides one-click-fix solutions and lets you define custom code quality rules using AST graph patterns, catching subtle issues that traditional static analysis tools might miss. CodeRabbit has so far reviewed more than 5 million pull requests, is installed in 1 million repositories, and is used by 50,000 open-source projects. Try CodeRabbit free for one month at CodeRabbit.ai using the code Pragmatic. That is CodeRabbit.ai and use the code Pragmatic.
Let's go into the specific details. We're talking about live streaming. Can you explain to us how does video streaming work behind the scenes at an architectural level? Sure. So let me share my screen to talk about how it works. Just give me a sec. So there's something called a source feed. This source feed is coming from venue. So from venue, actually source feed comes afterwards. From venues, we have...
feeds like camera uh so like the venue is in this case like the stadium where the matches play right yes and then you've got a bunch of cameras there yes so all the cameras are actually connected over fiber all of that come to pcr uh pcr is a production control room where like all the feeds from stadiums are coming and then there is a director so it's like a movie production where there's a director there's a producer there are people who are operating the cameras the director is telling uh to the uh to the uh to the person on the on the ground that which camera to use what angle to show and which uh which feed and so on so all of that is happening over there Lots of people, lots of yelling, lots of screaming. That's a place where we are not allowed to go in if it's on here. Nobody's allowed to go in over there, to be very honest. From PCR, we get us something called a source feed. This is basically what the actual production feed is. Now, this source feed is something which comes to... I'm talking about the video part of it, then I'll come to the server side or the backend part of it. This source feed is then fed into the... There's something called a contribution encoder.
This contribution encoder's role is to convert the source feed into a... Because the source feed is a very high quality, you need to compress it a little bit so that you can stream it to the cloud. And then from there, the source feed goes into the cloud, which is like a cloud ecosystem, like an AWS or a GCP or whatever, whatever the encoding is happening. This is kind of, again, a peer-to-peer private link. So it's nothing over internet, by the way. So it's like your master video feed, high quality, but already a little bit encoded or compressed to manage it. Yeah, source feeds can be in like...
hundreds of mbps or 150 mbps right the contribution encoder will bring it to a stranded profile of like 40 mbps depending on whether the event is being streamed in 4k or something like that from there it goes into cloud and within cloud there is your distribution encoder OK. And this is the system which is taking care of encoding it for HLS, Dash, or whatever formats a user is going to consume into. And this is with HLS and other formats, these are like formats that different players will be able to support, right? Yes, yes. So the output is like an HLS stream or a Dash stream. Different combinations of it, like you have a different type of stream for mobile, you have a different type of stream for TV, because again, the form factors are very different. The network on which they're going to operate is very different. So all those combinations we output, like in past, we've outputted more than 100 variants of streams to the CDN.
yeah just so i understand so you are transforming into multiple like you create all of these different ones right like you set up to 100 of them yeah so actually actually if you think about it uh there are multiple source feed by different languages okay so we uh we used to stream in like about 13 plus languages so there will be one source feed coming for each of the languages and with platform combinations and all of those the output feed gets into a range of like 500 plus kind of a number. This whole thing is becoming a lot more complex than you would have thought. Yeah, and then there's an orchestrator on top of it. This is where our engineering comes in. This is mostly whatever I'm talking about is probably using some partner technology and stuff like that. Then there's an orchestrator, which is basically controlling all of these systems, not at the piece, actually not at the venue. It starts from somewhere from contribution encoding to your cloud orchestration, what endpoint to push to, what should be the conflict of distribution encoder.
and what CDN endpoints to use. So all of this is managed by the orchestrator. So this is our engineering product, which is orchestrating because on a single day, we're not just hosting IPL. Like we used to not just host IPL, we used to host other events as well. There's a lot of football games going on. There's a lot of other e-sports events which are going on and so on. So while we are focused on the in-apply, but we also have other, we used to have other 50 events to take care of. So this orchestrator's role is to ensure that all of that workflows are set up properly. And from here, what happens is this orchestrator sort of generates a playback URL.
we now the orchestrator takes care of knowing what the final playback urls is going to be which is your playback endpoints okay and then once the feed and everything is started it will send it something called as a content management system this is what our users are interacting with so orchestrator will push all the generated playback urls over here And a playback URL, so is this like the endpoint of a node or a machine that has the right format? No, it's not an endpoint of a machine. It's an endpoint of the CDN where the video would be available. So for every TV, mobile combination of Apple, Android, and whatever variants exist, for each of them, we generate a playback URL. So a playback URL is a certain spec, right? And along with that spec, we push it to the content management system. Your client apps are generally interacting with your...
content management system for rendering the browsing experience right whenever you open the app you're seeing the content list of contents and everything so uh this is how it does but that's not that's not all right this is just for a browsing experience and typically to scale you would put put a layer of a good enough caching in between to ensure that things are uh scalable in nature right now like that uh that's the one of the secrets also of uh making things work now this is about browsing experience so far what we have spoken about is that you can open the app you can see the content you can click on the content you can see the details of the thing the moment you hit a play button you call a much more complex system, which is generally a playback system. I cannot talk about exact internals of it, but this system is responsible for ensuring that the user is authorized to watch the video. It will talk to a user system. It will talk to encryption and DRM systems. It will talk to the content management system and then give you a kind of an encrypted URL, which is what the client will use to play back.
And are we talking about live streaming, or is this playing back existing streams? This is for live streaming. The only difference in, like, whatever I'm showing in the diagram is all about live streaming. In our VOD system, you will not have this complex setup like over here, how you have the contribution code of cloud distribution. It'll be a little bit simpler. There will be a single encoder, which will encode and put it into the CDN. And there will be a similar orchestrator for VOD content, which will take care of this. So the workloads are almost similar, but there are more complexities on this part on how to select which you are to play back in live streaming than on a VOD system.
Yeah. One question I have is you mentioned CDNs, but CDNs are great for caching stuff and making sure, you know, spreading it out so that they're on the edge. But this is a live stream, right? Like every, I don't know, 100 milliseconds, you will have new frames arrive or every, I'm not sure how that's done. But how does this square with the fact that, you know, like CDNs are great for caching, but they might not be the best for real time stuff. Or are they good for real time? So they are actually. See, the secret sauce of how live streaming works is in the HLS and the Dash spec, to be very honest. I can talk a little bit more about HLS so that you get a sense of how it works, right?
uh see hls you have something called as master manifest okay so whenever you see a playback it will it will look something like master.m3u8 okay in this master manifest you have uh multiple informations so i'll so it will contain information of saying that this is a 240p video and this is it will point to a child underscore 240p.m3u8 and so on then similarly there will be a this 480p so on right so you would have layers layers of manifest right and this is a so what happens is within each of the child manifest you would have a list of segments OK. Segment1.ts. It is a segment of a certain time period. Yes. Yes. Yes. How long are segments usually? Yeah. So that is configurable. That is all subject to how we design, how we look at what kind of latency do we want to offer to the users on the existing. So typically, they are between four to six seconds is what widely used in the industry. So what happens is four to six second files, right? Yes. You have one after the other. OK. So what the client is doing is, essentially, in Enchilist protocol is, when you initialize the video, it calls the master manifest. Then it gets the list of all these child manifest. And it is keeping track. So based on the start bitrate.
based on the network condition the user is, it will pick saying that, okay, this user should start from 480p or 720p or so on. Okay. And then the child is just calling this manifest again and again. And if your segment duration is dead to four seconds, it will call the child manifest every four seconds and asking for an update. Okay. And the update will be the next file, the next, you know, part of the next segment. Yes. So this is like a windowed manifest, which has like 200 plus segments. Typically, you select a duration, which you want to keep in the manifest. And the player is kind of pulling this manifest and getting the list of segments. So as soon as it sees a new segment, it will go to the CDN request and get it down.
get it downloaded and put it in the buffer of its own. So that is how it works. So the role of the CDN is that the CDN is not operating at 100ms is what I wanted to call out. CDN is operating at like a four to six second gap. But still, with so much of caching, it becomes tricky. And this is where our engineering works is how we tune and fine tune the CDN configurations. What is the right TTL to use and so on. Because if you use too short a TTL, then everything is expiring and you're always missing a cache. And if you use a very long TTL, then you have a chance of hitting in stale data.
Yeah, so it's a fine game. But now, I guess, I now understand, you know, when there's a big game happening, may that be the World Cup or something like that, and you're watching it on the stream, every now and then, you hear a massive scream from the neighborhood. And then two or three seconds later, you know, you see the goal or see the team scoring. So I guess this explains, right? Because there will naturally be roughly up to a few seconds of delay, depending on which endpoint you're hitting, depending on when you started the stream. Yeah, good that you brought in this, right? I'll explain. So every time, any encoding system would be more efficient if it has a look-back period, right?
So if an encoder wants to be more efficient, compression technology is like saying that, hey, I have a reference frame, and these next frames are delta from this reference frame. So for compression to be more efficient, if you can look back more and more, you will be better, right? So there's something else called as group of pictures or group of GOP, right? You say that my GOP is two seconds or my GOP is one second or four seconds, which is to say that, okay, I'm going to look at optimized compression for this. So every encoding stage will add to that latency, which you're talking about. So that is one place where your latency is added. Then the other latency is added on the client side. Now, when you're streaming on a internet versus a TV,
In TV, what will happen is that if you miss downloading a segment like in segment EQL and certain frames, you'll probably see a blackout and you'll move forward. TV is not going to let you be, it's not going to optimize for you to be at the right place, right? It is always the latest point you're seeing because it's real time. It's real time, right? So if you miss something, you miss something. Like if there's a rain and you miss to get that packet on the TV, you'll miss it. But in internet streaming, every user is maintaining a buffer, right? And to ensure that you get a smooth streaming, you will always have some buffer configuration on the client side, which will be taking care of...
like your smoothness, right? So what happens is whenever you start a playback, you're always starting a playback from five seconds to 10 seconds behind the light point. Because if I keep you on the light point and there is a problem for any reason, right? It's a distributed computing, it's a complex system. For any reason, if you miss out to download that particular segment, you will end up seeing a rotor and you'll have a bad experience. So it's a fine choice when you're doing it about scaling versus number of devices you want to support versus continuous, like a better UX versus a more light point, right? Like we found a sweet spot somewhere where it works for the users. Again, I can disclose the numbers, but that's the complexity behind it. Yeah, but I really like it because in the sense of like, okay, well, this is the reality, but you know, this is what it's about, right? Trade-offs. Like you can...
you can choose between having, you know, like some latency, okay, five to six seconds, but then it works pretty smooth for most users and, you know, you don't have crazy infrastructure requirements. Or if, you know, our requirement was, you know, like, I don't know, someone came up with, like, you shouldn't have more than 100, 500 milliseconds, you would need to architect a lot bigger system, a lot, you know, different trade-offs, right? So I'll give you an example, right? Let's say you have four second segments, okay? And you've got, so you're polling for, you're downloading 15 segments in every minute. You're downloading 15 manifest calls. You're making 30 calls to the CDN every minute. Now if I change the manifest, the segment duration to two seconds, your volume of calls will double, which means,
even though you're downloading the same amount of data, the number of requests which will be hitting, the number of compute cycles which will trigger on the CDNs will increase. And no matter whether we say the internet and cloud is infinite, it is not really infinite. There is a finite amount of space and resources and everything. So when we're designing for scale, you have to factor all of this in from the perspective of capacity, from CDN capacity. So sure, your networks have gone to 5G and all of that, but there are so many choke points within the infrastructure that you have to account for, you have to be aware of. And that is where the playback system becomes very crucial because it is aware of all these nuances and deciding the best experience for the user.
Yeah, and I guess this just shows how different internet streaming is versus traditional television. You know, like when you had an analog signal from the radio go out, you send it one way, every device catches it. If it catches it, it catches it, it misses it. As you said, or when you have more traditional cable, you know, the cable connections to TV, again, you just push out the signal. It's the same real-time signal. So it just doesn't have this level of, you know, you don't have to worry about the internet infrastructure, the internet providers, your bandwidth, all this changing. It's fascinating. I do have one question related to this. Adaptive bitrate streaming is a word that I've heard, and, you know, I can kind of imagine what it means, and you kind of already touched on it.
How does it work and who controls the Adaptive Bitrate? Is it the server? Is it the client? It's a mix of it. Largely the capability is driven in the player, but it's all about fine curing. Adaptive Bitrating is, if I were to simply explain it, you have a network speed of 4 Mbps. Every layer which you see within the video, which is 240p, 480p, every layer has an average bandwidth. So 240p would be probably at 200 kbps, 720p would be at about 10 Mbps and so on. So what your player is doing is, as and when it is downloading that segment, it is also measuring your bandwidth.
Because you're downloading a 4MB of, let's say, a segment chunk, how much time it took. So it knows an average speed, right? And when it sees that it is drifting apart, like your download speeds are getting slow, it will recalibrate, and the player would jump to a lower layer on this thing. So it goes to the manifesto and says, like, hey, can I have the 240p? Because I can measure that. I'm not fast enough to download all this. Yes, exactly. So that's where you see the blurring of the video and all of that seamless switch is taking care of the player. But that is the simplest implementation of the exo player or the client type player now. Now, if you do that, and you try to run an IPL match on that, I don't think it will work.
You'll encounter numerous issues. You'll get a bunch of customer complaints and so on. So there are a bunch of parameters which the player allows you to control. Like what threshold to choose to switch between layers. What should be your starting bitrate? What should be your threshold for error? What is the buffer duration which is left before you make a decision to switch?
So all of those parameters are something which is fine-tuned when we are doing engineering. And then some of these parameters are governed from the server as well. So the server is also looking at how things are operating around the world in the entire ecosystem and making some calls. So if you have to do degradations, the server can choose to do degradations by limiting the number of layers you're accessing it. So what you're saying is when my server is getting overloaded, you also have the choice to start serving either longer segments, lower bitrate, et cetera, in order to make these decisions.
the place where I was saying that I'm pushing to CDN are numerous combinations. There are some combinations that are designed from a reliability perspective that we can offer the best experience up to a certain concurrency, but after a certain concurrency, we might have to switch users to a little bit more degraded. Yeah, it makes sense, right? Yeah, so all of this is now part of the engineering which we used to do. This is not something that CDN offers, right? And so CDN is, sure, handling a lot of segment delivery and stuff, but orchestration is where the tricky business is.
Yeah, so I guess it's safe to say just, you know, there's that thing where you're streaming something, may that be a live video, either live video or video on demand. And when it gets blurry, I guess two things can happen. Either it's your client's side deciding your bandwidth is not long enough, or if it's a live event, it might be that the, you know, the engineering team or the system decided that in order to sustain all this high load, it needed to switch back to conserve the server resources. Yeah, so the third aspect of it, right? Your client, now this is where your equilibrium kind of concepts come into the internet, right? When, as and when the congestion starts.
lets you connect you to a tower. Now, 100 people are watching earlier. The tower is a finite capacity to stream downstream to, let's say a- Sorry, by tower, you mean like a mobile tower? Mobile tower. Like a 5G tower. 4G or a 5G tower, right? They have a finite capacity on the number of clients they can handle, like an access point. If they start to get congested, they will start to naturally throttle you or start bandwidth sharing, which will also trigger slow download. So this is not a parameter where I am going and controlling and saying that, hey, the user should get blurry or this thing like that. It is natural which will happen, right, depending on the infrastructure layout and where you are connected. There's so many layers into, you know, what-
We started off as just, you know, like, oh, just stream live video. So it's funny, right? When we get user complaints, you have to look at so many places. Like, was it our system? Was it our client? Or was it like some intermediary network? And if it's intermediary network, how do you prove that there is a network condition? Because we do not control those systems. We do not have visibility on the metrics of those systems. So we have to build intelligence on the client to figure out all of this. Yeah, it's...
It's very complicated. When I used to work at Skype and, you know, it was on my LinkedIn, people knew. And sometimes people would either message me or a friend of a friend saying, oh, you know, you work at Skype. So I had this problem where I had this video call and it got really blurry. Like, why was that? You know, like you said, I didn't have as deep understanding of everything that happened. But you mentioned that you cannot monitor everything. What is it that you can monitor? Because I imagine, you know, just as an engineer.
You need to have as much information as possible to figure out what is the best experience, but there's going to be practical limits and also just data collection limits. So what is it possible to monitor and what is practical to monitor on the client side and then on the server side? So the amount of data collection processing you can do is infinite. There's no end to data collection analytics. What it comes down to is what will help you to, in case of an issue or an incident, what will help you to figure it out? And the way I describe metrics is that there are leading indicators and there are trailing indicators.
Okay, leading indicators are things which will ahead of time tell you that there's a problem about to happen. So we would have three or five metrics which are defined as leading indicators, and we would be very sensitive towards those metrics. Like on the client side, it would be the amount of buffer time you're seeing, which is how many seconds did you see a buffer in a minute, or what is the playback failure rate, which is basically when you start a video, did you encounter a failure, or while watching a video, did you encounter a failure, which is a fatal image. Like not the app crash, but the video crash kind of a thing. So there are some of these metrics which will be treated as a leading indicator.
And these metrics will be collected on the client side. There will be corresponding metrics on the server side. Obviously, you'll be wondering bandwidth, number of requests coming in, latency, response times, a bunch of other things which you'll be wondering on the server side. The leading indicators get priority over the traffic. Like you would always want to have them process ASAP within a minute, sometimes under 30 seconds also, so that you're reactive, you're getting alert before anyone complains to you. So that's where you optimize for. Then you have trailing data, which is basically, let's say you figured it out that there is an issue. Now you need to get into the details of it, right? And the funny thing with the live streaming is that
your time is also moving right it's like it's just not that okay there's a system state or there's a state right that the state is also constantly evolving so you need to measure a lot you need to collect a lot of data and you need a lot of tooling to be able to visualize uh what is happening and how it is being rendered so so there's a lot of data collection will get processed a little slowly it's available on dashboard for us to uh consume and figure it out but uh that's how we look at metrics and then on the client side obviously you're collecting all of this data but then you need to like push it back to the server right like every every you know again i'm assuming this is an entering trade-off like if it's every second every 10 seconds every minute that kind of stuff I don't think every second would scale for us. 30 million users watching concurrently and every second they're sending data is like, I cannot scale that system. Yeah, but it's interesting because now we have clients downloading and then we also have the upload of the metrics, which is also this fine balance, I assume. When you decide, okay, even just this relatively simple example of how often should a client upload their leading indicators?
How did you decide on what that will be? May that be every minute or so? Did you go to whiteboard and figure out what does this mean for a server? How much can we process? How many devices? Was it a mix of prototyping and doing load tests? Or was it a combination? It's a combination. It's not an easy decision. In fact, during the event itself, the frequency keeps changing. As and when you're scaling, you'll be okay to compromise on certain aspects to scale the system and have some sense. You will also use some form of sampling as well. You'll use a balanced approach to figure out how to do so.
You can collect all the data at every second level, but you have to process. There is cost associated with it as well. And just not cost. Again, cloud is finite. So even though CDNs are finite, even your cloud service providers also have finite capacity. So you have to take into account that do you want to prioritize your playback systems and content management systems, or do you want to prioritize data collection? So all of this has a priority and an association to say what tier of the system falls into, and that tier design defines how the system degrades. So there's always a degradation framework, which is to say that, okay,
If the match is not huge, then yes, we'll collect more optimistically at every 15 seconds or every 30 seconds or whatever that number is. But as the game starts to become hot or more and more traffic starts to come in, then you would at runtime start to degrade systems, which means that degradation happens in different shapes and formats. You will probably reduce data which is coming in. You will change the interval at which you're collecting data. You would start sampling. All of those parameters start to come into the play. And all of this is part of the design.
This is all the exercise we do before the game on resource planning, on capacity planning. Like capacity planning is a huge exercise. And that also decides what are the cutoff points, right? Sometimes you will say that, okay, I need to support up to, let's say, X million users or 50 million users, 100 million users. That means you have to work backward from there and say that if I have to support 50 million users, what are the P0 service requirements? And then you say that, okay, this is the residual capacity. So the P2 services have to operate in this residual capacity, which means that I can do certain things. I can't do certain things.
So on capacity planning, you already kind of outlined how it works, but you said it's a big exercise. What does a good capacity planning look like? And then, you know, like, why does it take so long? Because, you know, what you described here, honestly, yeah, you know, just do the maths, do this, that. But obviously it's more complex, right?
So, capacity planning is a very complex exercise. In fact, for most of the video streaming, the capacity planning for the next year starts at the end of the first, like the previous year. Because you have to lay down infrastructure, you have to work with providers to ensure that their data centers have to scale up, their network have to scale up, their power requirements have to scale up, all of that has to be taken care of. It's not that simple. So there's physical infrastructure being involved, right? You're not, you're not just, like, providers will not go and scale their infrastructure. Sure, they have a 10% plan, 20% plan, but if you're operating at a much more intensity and adding more year on year, then you have to work backwards from there. So that is one aspect of it. The other aspect is,
during the tournament, there can be overlap with other events. So there might be reservations done by other companies. Like there's a big e-commerce sale going on. Let's say there's a Black Friday. Black Friday is not there. So the point is you have to look at the resource. You have to figure it out. You have to ensure that you've blocked enough resources for the time period. And the other complex pieces, how much to ask.
every year the nature of the game changes there is no set pattern how many people will come in so there are certain models we will look at it and see that okay this is the user traffic which has been coming in which was in the previous year and that's we have seen platform to grow by x percentage so we'll apply some modeling to it and figure out what should be the next year's optimistic number Actually, not an optimistic number. It's a very pessimistic number. We operate with a very pessimistic number. And then work backwards from there. So the reason it's complex is because you have to work with so many providers, understand how much resourcing and all of that is there, which means they have to go back within their engineering teams, get that numbers, share that number with us, and then we do others. Just to make it really concrete, are we talking about how many virtual machines you'll need, how much bandwidth you'll need, what are the things behind the resources that you're going to have a list of, oh, we need this much?
Primarily, it boils down to the same core system metrics like compute, RAM, disk, and network. All of that boils down to that. Every tier has those numbers which you have to figure it out. When you're running servers, it's going to be number of vCPUs. Generally, that is a limiting factor. Disk and RAMs are not limiting in cloud. Network becomes an interesting problem for the streaming part of it. Actually, it becomes interesting on the API side also. Because, yes, video you're doing...
like tens of TVPs and probably more than that. But API is also consuming a significant amount of data download. And those networks are designed differently than the video networks, because video networks are more designed from a heavy caching perspective, that the offloads would be super high. But APIs is mixed traffic. There's security. There's different tiers of security. There is PIA data flowing in. There is your firewalls and so on. So you have to look at all of those parameters. When I say parameters, again, it's network consumption compute resources available at each hop, figure out what is the least common denominator, and work from there. Yeah, and you did mention before that the cloud is
finite now i didn't hear too many people saying this because usually when you you know like i'm a startup i'm building something that like i will be aware of the cloud is expensive but i'm never gonna think that it's finite because as far as i'm concerned i can always pay more to you know scale up in your case that's not really the case right so and this is because it's so you're so large right this application is so large yeah the use case is so large that uh like year on year the growth is so much that it's very hard to preempt right and you're as i said that you're doing pessimistic planning because you want to support best best of best uh the infrastructure exercise the the growth of infrastructure is relatively hard because I'll give you an example, right? So let's say in a city like Bangalore, Mumbai, there's a finite number of TVPs available and the capacity is available. Now, if you have to add more capacity, the providers need to purchase real estate. They need to add links to it. They need to deploy servers to it. The servers need to be procured, imported from another country. They need to be configured, onboarded as part of the network, and so on. So it's a slow process. It can't happen overnight. So basically what you're saying is, let's say you as a geo or any other streaming service is like, OK, we think that
This game that's going to be in six months. Last year in Bangalore, 5 million people watched. I'm just saying a number. This year, we think it'll be 8 million because of our projections. We think the current CDN capacity based on our calculus is not sufficient. So now we're going to go to CDN providers and try to make a case for them to invest. That's pretty hardcore. Not even engineering. This is like, you know, it goes beyond engineering. Yeah. And you started with this question saying that, hey, why don't you code? Because this is a much more harder problem than actually coding it. Wow. Okay. I'm starting to understand. It's important to stay connected, but it seems like this seems...
It's probably more important to do this thing and go to the vendor, sit in a meeting with them and explain to them why you need to order those servers from wherever. Another common question I'm asked on the subject is that, hey, oh, sure, why don't you go and get it from Delhi? It's not that easy. If you understand, it's a caching infrastructure, it's a tree, right? It's also edge, right? Yeah, it needs to be closed. The more upstream you go into a branch and try to get it, the capacity available over there is finite. So if I were to say that, hey, let's root this excess of 2 million traffic, let's say that's under 2 Tbps.
There has to be that amount of backbone infrastructure present in the country to be able to rotate to another city. This episode was brought to you by Augment Code. Wish your AI code paddled actually near your code base? Augment Code is different. Augment is built for professional engineering teams working on complex systems, not hobbyists building to do apps. Other AI assistants struggle with context. Augment understands your entire code base, architecture, patterns, dependencies, and all. We're talking instant understanding of systems with millions of lines of code in under 200 milliseconds. Teams like Webflow, Lemonade, and Kong use Augment to ship better code faster.
No more context switching, no more generic suggestions, just an AI that works the way expert developers do, fast and in flow. Plus, with enterprise-grade security and zero training on customer code, you can trust Augment with your systems. Start building with Augment in your favorite IDE and see what it's like when an AI actually understands your code. Try it free at augmentcode.com slash pragmatic. So we already touched on this. This is, I think, one pretty good example of like an India slash APAC specific engineering challenge. What other engineering challenges have you seen that are pretty specific to APAC? You know, like if when you're building a system in Europe or US, maybe you don't even need to think about it. But in your case, you know, I gave you a lot of headache.
Capacity is one aspect of it. The other comes is the mobility. India is a country which is mobile-intensive. People don't watch it on TVs or laptops or this thing, right? India is a country which kind of had a leapfrog in terms of generation, right? People who are not using any computers or any mobile devices started onboarding mobile devices directly. They did not see laptops or tablets or connected TVs and so on. Connected TVs is still picking up, but it's still not there at the level at which it might be there in Western countries. With mobility comes the problem of that, one, the device is constantly moving. It is changing radios and networks. Oh, yeah. And you're changing cell towers and all that. Cell towers and all. A lot of audience are...
like the taxi drivers or the cab drivers or people who are on the move who are getting off the office. The event happens in the evening, people are getting off the work. Yeah, everybody's holding the phone and watching it. Exactly, right? Like most of the ads used to run like that. So people are watching on the move. So that becomes tricky. Sometimes you're connected to a 5G tower and then you go to a 4G tower and 3G tower and so on, right? Like you're on a train and you're moving at a speed, your towers are switching. That is one aspect. The other is a battery. Most of the games happen in the evening, right? And if you started your day in the morning, you probably charge your phone, you're off the work.
You're not getting it out to your battery. So that becomes an interesting problem to also tackle, right? How do you ensure that you're not doing so much of computation on the device that you end up burning up the phone battery? Oh, wow. I would have not, this would have not been on my top list of things to worry about, but I guess there you go. You need to worry about it. You need to worry about it. Like, you're streaming or your downloads or video content cannot eat up the phone battery, otherwise the user will not watch. And how, what do you do around this? Like, what kind of considerations do you have? So it's not of how much background processing you're running, what's the phone brightness you're using, what's the volume levels you're using.
what is the color intensity, all of these parameters, the profile of the video kind of controls how much battery consumption is going to happen, how frequently are you polling, how frequently are you downloading, what kind of complex encoding algorithm you're using. So we're using H.264 versus H.265, the codecs, right? Like HEVC versus AVC, each of the codecs have its own complexity. If you use more compression, then it means it will use more compute, which means it will consume more battery. so so do you have this thing where it's theoretically possible then to have on the client side based on the battery level request a different type of stream you could do that we could do that but uh the point i'm trying to make is that all of these parameters are taken into account when you're these are not probably decisions that you make at runtime because at runtime you would be more focused on like you would want to have as less variables as possible uh so this these would be considerations that you'll probably do when you're releasing features or when you're adding more the same thing when you're building it yes when you're building it because runtime like I would want to have only one or two parameters which I'm tweaking at runtime. More than a number of settings, it requires more coordination, verifications, testing, and all of that. Sure, you can have automation, but trust me, automation doesn't work after a point. You need to still have some level of manual testing. Okay, so...
an event is on and you know it's you know it will be a big event and you know more and more people are joining in and you know the load just keeps going up and you know eventually you set a world record but as this happens what how can you scale things up how can you both plan to scale up a system and what can you do on the spot and i'm assuming horizontal scaling i'm sure will be a thing but for example is vertical scaling basically having bigger boxes bigger devices also an option or a practical thing to do so uh i think a vertical scaling See, vertical scaling only happens on the database or the database layer, to be very honest. Computers are largely horizontal in nature. We try to, like, for example, if you're using a... Actually, databases have also now become horizontal in nature. You can add more boxes and you'll start to split and scatter data. But you would want to avoid any kind of that, like scaling at the data layer, at all when you're operating at load. So most of your databases and caches are preemptively scaled, pessimistically to a higher number. Compute is what you're controlling on the fly. In fact, autoscaling doesn't work.
uh so you have to always auto scaling doesn't work for this kind of workload okay something more i i would have thought you know the whole point of auto scaling is uh that it should work but obviously under it it's a bit complex can you can you like give us a bit of a you know explainer 101 sure sure so why does the auto scaling not work yeah see the auto scaling works by uh adding a box at a certain rate right and then default auto scaling which is provided most of the providers have a like a cool off periods and a bunch of parameters which is not entirely flexible so what happens is that when the game starts or when there is an innings break and the audience come back that's a sudden search And in our experience, the autoscaling has never been able to respond to that in a very speedy manner. So let's say before the, so I'll give you a class example, right? Let's say before the innings break, you were operating at a 20 million conference, I'm making that number out, right? Now, innings break happened, 15 million people chose to drop off, 5 million straight back. Now when the innings recover, you would have 20 million people coming in. But the autoscaling is not going to respond with that number in mind. It will be like, oh, I'm seeing this much traffic, let me add more boxes, more boxes. But you have to understand that there were already 20 million people which are on the stream. So after the innings break, you have to go back, right? So if you use autoscaling, you'll be screwed.
So that's why we used to have our own custom scaling systems and custom scaling providers. And that will look into a concurrency. Concurrency is our metric which we would align to and say that this is a golden metric to scale against. We will build models to translate that to every system services. Because it's easier to talk in that language than actually talking in another language. Every system having a different metric to scale against is complex than saying that everyone in the company is scaling against a common metric. And we are very solid and sure about this metric. So you can compute and say, okay.
I have a number of concurrent users, which means that this is the traffic I would expect on XYZ services. This is my user journey. 100 people open the app. Y number of people go on to the playback page. Then Z number of people open this. So there's an entire formula which you can build out. And then you can, from there, map back and say that, okay, this is the amount of requests you would see. And there will be certain systems. I'll give you an example. When the innings break happens, again, innings break or some kind of an event happens, let's say a key baller or a key batsman get out, right? What people do is people have a tendency to press back button.
the moment they press back button press back button you mean the exit stream or they want to just rewatch it again no they just want to go back like like close close the stream and go back to the home page oh gotcha exit like either people will just swipe up and close yeah but a lot of people will press a back button the back button will take these users to the home page and home page has a different set of apis to be caught which was not seeing the traffic at the rate it was seen before and they're gonna have a big big spike exactly so so all of that intelligence can't be taken like answering your question on why auto scaling doesn't work is because of these ux's and user journeys which which cause problems so basically you're saying the whole the whole company you know or the team will know that this is our concurrent user target and every system just needs to prepare to potentially in a single second get that many you know requests if you know people like in your example is hitting the back button and fetching the home page yeah so so so we have custom scaling and so all of this isn't present so that's how the systems are taking care of these custom scalers are constantly watching at the concurrency number Whether the concurrency number coming and being reported is accurate or not is a big question. So you need to have triangulations and proxy to figure out whether the concurrency number coming from the system is correct or not. Because if that gets messed up, your entire scaling gets screwed. So you've given us already a very deep dive into...
video streaming and we probably just only scratched some of the surfaces but how how did you learn all of these things outside of on the job are there additional resources that you do or you kind of educated yourself basically if someone wants to understand a lot more about live video streaming what are recommendations you would you would give them and you know and they don't have the opportunity to work at a company like you did just yet yeah so this is i think it's a nice part and the sad part you cannot google our issues there's no stack overflow for our issues of the things we experience to be honest Some of the things around the video, there's plenty of good resources around the video. There is a GitHub link. I'll probably find it and share it with you. That is a very detailed explanation of how video works. It starts from text and then goes on to how image is done, how image compression is done, and then how image becomes a video frame by frame description. There's a very good document around it. And so I can share that. So that is something which I learned. The rest of all of the stuff is on the job. I don't think these problems are something which are available on the internet anywhere. On the job, what were things that helped you become a lot more experienced, reliable, senior?
What kind of approaches or mentality helps you to soak it in just really quickly? So I think for me, it has been planning the entire exercise of... going through and drills. So we used to run something called as a game day. Now, this is basically a simulation of how an actual match is going to be. So you were simulating this highest traffic event ahead of time? And not just traffic. We are simulating the entire operating protocol. Oh, wow. Really? You did that? Yes. So we'll say that, OK, the match is going to start at 7 a.m. 7 p.m., let's say. Now, every system is supposed to scale beforehand. There's a timeline to it. There is a checklist to it, and so on. No matter whether the teams are ready or not, we will start the live stream and start sending traffic. Because in a live event scenario,
you're not you do not have a green light you're not carrying a flag to say that okay the train or the traffic can come in right the game is going to start at 7 30 p.m it's going to start at 7 30 p.m no matter whether you like it or not so we used to do those kind of a drill and in this drill just so i understand were you simulating you know like synthetic traffic or was it more that every team had to scale up you know like do whatever we would generate like we would generate terabytes of uh yes we would just there was a little spinning up you know like pretending to be users so we used to have a framework to simulate that kind of a traffic There's a platform called flood.io which we used to use, which will actually credit. And we will not give teams access to that dashboard. Yeah, so they wouldn't know what's getting them. They don't know what the traffic is going to come, what kind of a pattern is going to come, and anything. Because that's the exact set of production. You don't know what is coming your way. You need to have your systems being able to tell you whether they're in a healthy state or in a bad state. So all of this is like learnings from there, learnings from there. And most of the learnings like...
I've been into this industry for about like seven plus years, right? And it's not like we reached that 32 million number in a single year. We've done smaller numbers and figures. So over the course of time, also you've learned and while building the systems, we learn from that. Well, and one thing that I kind of, I appreciate that you told us is, you know, I was very focused. I think everyone's focused on like, oh, when you set the record, but you said something interesting, which was it was a 70 day marathon and the beginning was actually a lot more challenging, you know, and it wasn't setting records, but it was already the traffic patterns, the, you know, dealing with all of those things. So I think it's a good reminder that, you know, like sometimes it's not the thing that gets the most publicity that is the hardest work. It's the work that you put in before.
no the first the first week so it's a 70 to 80 days by the way not 70 days wow we're not following cricket and you know we're just used to you know either football or other games where it's just a lot shorter it's a very long time period it's a very long time and it can get very intense on people as well like it's very stressful every day there is a match some other days there are two matches like back to back and two matches a whole complicated scenario because how do you switch from one match to another match traffic shifting and all of that becomes super complex it's not that straightforward from an operational point of view so Yeah, the first week is very intense. The days before the first week is very intense. But after that, you start to settle in. We also try to aim that every year on year, we have improved on our operational protocol, automated a lot more. So I think someone's telling me that, so I moved from Hotstar to Geo Cinema, right? At Hotstar, the level of automation that pretty just like people can actually be on a coffee and the live stream match up to 20 to 30 million match and just run without people.
Well, you know, over time it gets easier. Yeah. So for engineers who would like to get better at architecting systems, what were things that helped you in between, you know, building them, coding them, being in these meetings? Were there, you know, and I'm talking not really about live streaming. Now this is more like generic software engineering. Would you have advice on activities they can do, behaviors they can pick up that helps you just learn faster?
So there are certain things which I can share what I do and probably if that helps. So I think one of the things around scaling and building complex systems, which I've learned the hard way, I would say, is that you must understand anything and everything which can fail will fail. Murphy's Law. Okay. people are very overconfident about their systems. They're like, hey, we have done everything, we have scaled databases, we have done transaction systems and all of that and so on and so on. They underestimate. And I can tell you, I have countless examples of where people have come to and said, hey, our systems are scaled up, everything is fine tuned, this is good. And I have broken their systems and load testing.
You're good at making friends, I can see. Yeah, so people don't like me. No, I'm just pulling your leg. Yeah, but I'm just saying that that's one core principle which I feel is super important. The moment you get deeper into that principle, you start looking at everything within your system, every decision, every configuration, everything, because you know that that configuration at some point can miss. Even the smallest thing as database connection pool sizes, the number of connections you're making to database, the number of servers. Like, for example, someone said, hey, our connection pool sizes are fine, we can scale infinitely.
But there's a number of database conditions you can make through a database. So when you're scaling compute, this compute can scale up to a finite number because if it goes beyond that, it will start to open databases. Then your requests and queries will start to get queued up, which will increase waiting times and so on. That is one. The other is metrics and measurement.
it's very important to have very detailed measurements in place for your systems to be able to figure out where the problem is. Kind of a water flow and the same. There's a common paradigm problem I've seen with every engineer. So you know how you measure response times. Now, people go and measure response times by looking on the load balancer that this is my request and this is the response time. Now, even in the load balancer, there's a request. The way to load balancer is, there's a request queue. The request gets queues and then there's like a...
cron or like a worker which will pick up the request and process it. Now, the way you measure response latency is from the time you queued the request to the time you return the response. That is your response latency. Now, what people will do is people in the processing function, which is listening to the request, will put their timing function to start of the processing function and end of the processing function, which will get response latency. But they're like, oh, my server is working all fine. Everything is good. My response latency is sitting in the queue. It's sitting in the queue. And this is the same problem which can happen down the track. So that is the level of detailness you have to go through when you're designing metrics and measurement. Yeah, and I guess what I'm hearing is it also really helps for you to just
don't you know like go deep and understand exactly how it works because you know for a load balancer like it's kind of easy to go which is balanced a little bit once you go in there and understand like how is it implemented what kind of queue that it does use how big can it be all of the internal latency etc yeah thank you yeah so that's my third third thing is i hate apms i've never used apms in my life application performance management yeah yeah so a lot of people say that hey why don't we put apm we can figure out stuff on there now apms have it like at least this is my bias i feel that it makes you lazy sure it can help you get all the metrics and everything in place. But people become lazy. People don't get into the deeps of their system. They're not measuring. They're relying on APM to do the right things for them. And sure, APMs have evolved over the course of time. You can argue. I'm sure if this video goes out, people have a lot of views on it. But the point is that I feel like it makes you lazy. Not having APM forces you to go into the details, forces you to look into every corner, forces you to measure every aspect of your code, measure every performance of your code, and find your name. Well, thank you. And to close it off, we'll just do some rapid questions. I'll fire a question, and then you just go, what comes to it? What programming language are you the most productive with? And which one is your favorite?
So For building systems and services, I use Java. Otherwise, for scripting on a script junkie, I'm very familiar with Ruby, and Ruby is my go-to. For me, it's like English. You're writing English, and it just works. So Ruby is my go-to for any scripting work and stuff. Nice. It's a friendly language. I mean, most people will use it because of Rails or together with Rails, but by itself, it's a nice language, no? Yeah, I started with Rails, actually. My career started with Rails, but then I got hang of Ruby. It's very easy. If you have to process some data, you have to analyze something, you write some models, you just write in Ruby. You get all the gems, start working, and just get it. What are some blogs or websites that you read to keep up to date?
I read a lot of Hacker News. Hacker News has a lot of fun content and latest content over there. Then LinkedIn is a good source, to be very honest. Somehow, the number of people I have, they keep posting a lot of interesting stuff. So I get a lot of good content from there. Apart from that, I have keen interest on latest technology. So like CS events or some of the events which are happening, what are the talks being presented, what are the things being spoken about is something I keep a tap on.
Awesome. Well, this has been just very, very interesting to go into all the complexity of something seemingly so simple as live video streaming. Thank you very much for sharing all this and for being on the podcast. Yeah, thank you. Thank you for inviting me. It's been a wonderful conversation, sir. Thank you to Ashutosh for taking us behind the scenes of building a large-scale live streaming service and one that set a world record at the time. Check out the show notes below for references we mentioned in this podcast and to see related deep dives you can read in The Pragmatic Engineer. If you enjoyed this podcast, please do subscribe on your favorite podcast platform and on YouTube. Thanks, and see you in the next one.