China's Great Firewall, or GFW, is often spoken about but is rarely understood. It is one of the most sophisticated and opaque censorship systems on the planet, and it shapes how over a billion people interact with the global internet influences the design of privacy and proxy tools worldwide and continues to evolve in ways that challenge researchers, developers, and policy makers alike. Jackson Sip is a PhD researcher at the University of Colorado Boulder, whose work focuses on uncovering how national-scale censorship systems operate. Jackson recently helped elit a groundbreaking study analyzing a previously undocumented GFW technique that quietly broke fully encrypted proxy protocols across China for more than a year.
In this episode, Jackson joins Gregor Van to discuss how the Great Firewall works at a technical level, the 2021-2023 blocking event, the pop-count-based detection algorithm his team reverse engineered, the cat-and-mouse ecosystem of censorship circumvention, and what these findings mean for the future of the open internet. Gregor Van is a security-focused technologist, having previously been a CTO across cybersecurity, cyber insurance, and general software engineering companies. He is based in Singapore and can be found via his profile at van.hk or on LinkedIn. Hello, and welcome to Software Engineering Daily.
My guest today is Jackson Sip. Hey, happy to be here. Yeah, so we're going to be getting into a pretty interesting topic with Jackson today around what is called the GFW, so the Great Firewall, and we'll get into what even that is, but also Jackson's done a lot of research into how it operates and actually how it operates through the years because it's been changing as well. So before we get into all that, as we like to do, maybe just sort of Jackson, what is your background? Did you get into this kind of research in the first place?
Yeah, sure. So yeah, I am currently a PhD student at the University of Colorado Boulder, and I've been here for about five years now. I'm advised by Dr. Eric Wistro, who's been doing this work for always longer than that. And as I first started my PhD, I got involved with an organization called the GFW report that has focused on censorship within China.
And so through working with them, I've done a number of different projects exploring the relationship between China and censorship. Awesome. So maybe let's just get into what is, I always kind of get the acronym mixed up because I think there should be a C in there somewhere because it's like the Great Chinese Firewall, yeah, maybe what is the GFW? So the GFW is the Great Firewall that is known, tends to refer to the censorship mechanism in China. That acronym has sort of expanded as it's being popularly sometimes referred to Iran's GFW.
So we often refer to it as China's GFW now. And it's a collection of different techniques and deployments that are spread around China in order to filter citizens in China's access to the internet. It affects a number of different protocols. We see it on DNS traffic, on TLS traffic, on quick traffic these days. And it's also been used almost as a weapon against organizations or services that have been complied with the censorship, the desire of the nation.
So one of my favorite facets of the GFW is a tool called the Great Cannon, which is sort of uncommon, something that not many people have heard of. But it was this technique that they deployed in 2015 against GitHub, actually. So GitHub was allowing for proxies to sort of run through pages, I think it was. And the idea was, is China couldn't get them to comply with, hey, we don't want you to be this proxy source. And so what they did instead is they looked at all of the HTTP traffic, any crypto traffic that they could going to in from China.
And they would inject a JavaScript request in that traffic that would make a request to GitHub. Right. So effectively they designed the largest denial of service attack ever by injecting these JavaScript requests into every HTTP request that went across the GFW. Wow. Okay.
I think I had heard of GitHub play a part somewhere. But yeah, that's new to me as well. So I guess, like looking at the fact that you've done a lot of research in this space, again, before we get into what the research sort of turned up, why again, just for the audience, some people in the audience may have been to China, may understand roughly what happens from the internet perspective. Equally, I think let's just assume zero knowledge. So like, why is this kind of research difficult in the first place?
Sure. So the goal of the GFW is to restrict Chinese citizens specifically their access to information outside of China that may be desirable. Right. So this might be like Western entertainment that maybe goes against the goals of the CCP, the Chinese Communist Party. And what we really struggle with is finding ways to detect that censorship, not being citizens located in China.
Right. So the biggest kind of logistical issues we have is getting access to vantage points, right? Places in China that we can connect to and then send requests that may or may not get blocked by the GFW. Additionally, because we aren't part of the CCP or the Chinese government, right, we don't necessarily know when we've reached ground truth, right? I think the data says sort of a black box system.
And we can, to some extent, only speculate, right? It takes a number of experiments to really determine whether or not what we have observed is what we think it is or just some other effective network. Okay. So before we get into what the research was, there's also kind of like a time frame thing here, which is helpful. So something kind of happened in, I believe it was November 2021.
Let's start there. What happened then? And maybe the data is kind of almost kick off why the research was undertaken or you can help us out. Absolutely. Yeah, sure.
So I think that the instant you're referring to November 2021 was the deployment of a new technique by the GFW to detect and then block fully encrypted protocols. And I think to understand that, we need to sort of get back to how Chinese citizens have responded to the GFW, right? Which is there are plenty of people in China who disagree with it, right? They would like to be able to access maybe whatever is latest on Netflix or something else. And so they will use proxies in order to circumvent the censorship and connect out to whatever content they otherwise might be able to access, right?
And these proxies use a number of different techniques to be censorship resistant, right? If you think about a proxy that you've accessed before or maybe like a VPN that you've connected to before, that traffic, it's obvious, right? We can look at it and tools like Wireshark, right? Or observe it on the network somehow and see that that is clearly a VPN connection, right? Or some type of standard proxy connection.
So the GFW has designed ways of detecting and blocking that, right? And so proxy developers kind of had to get more creative and they had to develop protocols that would be less obvious to some type of network observer like the GFW. And these tools took two different forms, the first of which being mimicry, right? We are going to try and make our proxy look as close to all the other traffic on the internet. And if you aren't familiar, almost all the traffic on the internet looks like TLS because everybody's doing something that involves a website, right?
And so the first thing they did is try to mimic TLS, right? And look as close to normal TLS as possible. But with TLS kind of having sharp edges, there were always these issues that would exist. And the sensor could then block that traffic because they would see that, say, like, the encryption scene was wrong, right? They picked the wrong site for something like that.
And so this led to a different approach. If we can't look exactly like everything else, let's try to look like nothing. Let's try and blend into the background. And that led to these protocols that we've referred to as fully encrypted. And a fully encrypted protocol involves some type of key exchange between the client and the server.
And then after that key exchange has occurred, every single byte that is sent from the client to a server and vice versa is encrypted. There's no protocol header. There's no exchange of information about size or anything like that, right? It's just going to be encrypted bytes as the payload of like a TCP connection. And the idea was, OK, this just sort of blends in.
There's nothing to worry about. They won't be looking for this type of traffic. But these tools became pretty prolific. They became the standard, right? And every proxy had sort of implemented their own version of this fully encrypted protocol.
And so in November of 2021, users all of a sudden just kind of accessed their proxies, right? And it wasn't just one, right? It wasn't just one particular implementation, but it was widespread. It was all of these different proxy developers, right? Shilis Ox and V2Ray are two names that are pretty popular in the Chinese censorship or image and space.
And both of their services had sort of fallen over as a result of this attack. And this was alarming because this had been a very stable approach for years. And users were trying to understand what had happened. And there was no clear answer of, oh, this one developer has decided to pull this site, things like that. Yeah, this is very interesting because on the layman site, I do remember there was just like a point in time when I was living in Hong Kong, yeah, I guess in November 2021.
And beforehand, and there was this sort of watershed moment where people started saying, oh, my VPN just doesn't work in China anymore. And it was sort of like, make sure you get maybe the quote correct VPN before you get there because you've got no chance of like getting it whilst you're there, that kind of thing. So it kind of became this which one works. And some people have even seen like posts today where like, I think people so go into mainland China and haven't fully anticipated the kind of tools if you want to call it that they'll need to just build access things the way that they would hope to be accessing them. Absolutely.
Yeah. Great. So let's move to the research that you and your colleagues did. I think it's like fairly groundbreaking research into sort of how this all worked. Yeah, let's just like talk through that.
I mean, I believe it was a sort of six month setup that you had to put in place. But I mean, like, let's kind of start there. Like where did the research begin and how did you go about it? Yeah. So as I said, I've worked extensively with this organization called GFW report.
They were sort of the leads on this project. They were the first people to be notified of what was going on. And there was no explanation to start, right? It was sort of this widespread event where no one knew exactly what tools were working, what tools weren't. And it started with effectively just seeing, OK, which of these tools have all the sudden stopped working and what techniques were they using, right?
And that led to this realization that, OK, it is in fact these encrypted protocols that seem to be the issue, right? If we send a regular TLS connection or TLS style proxy connection, there doesn't seem to be a problem, but the blame for traffic is an issue. And so what we ended up doing is setting up a number of VPSs in China, primarily in the Tinsen and Baidu clouds. And then we set up some sync servers in the US, right? So we had a number of different university partners, Boulder, as well as University of Maryland and the University of Massachusetts Amherst.
And we set up servers at all of these universities that we'd just sort of listen any TCP traffic that came to them. Nothing particularly interesting on that end. What it allowed us to do is we could now send traffic from inside of China outside to the US, right? Crossing that border and effectively triggering any censorship that may occur there. And while this sort of set up our architecture, we didn't sort of had to go into, OK, how are we going to determine what exactly is fully encrypted traffic, right?
This is a hard question. And so what isn't encrypted traffic and how did we figure it out? Well, the first thing we did is started sending in these TCP payloads just random bytes, right? See that this is getting blocked, OK? That makes sense.
What if we send all zero bytes or all one bytes, right? And I'm sitting in my advisor's office late at night. We're trying to figure this situation out and we see that certain bytes would get by when others win it, right? When we just sent a long string of the same byte. And ultimately what we found was that they were determining the entropy of the payload based off of the number of bits that had been set in the total payload, right?
So they were doing a count, what we referred to as a pop count on each byte. And if it was roughly 50% of bits set zero and one, then they would consider that to be high entropy traffic and then they would ultimately block those connections. So we're going to get into pop count shortly. Just kind of just thinking on, I guess, the setup here, whether any kind of risks with this, I don't know, did you ever think that, like, effectively within China, something would get triggered and sort of there might be any risks associated with this? Or is that just something that comes along with research and you have to just accept that that may be the case?
Yeah, that's a great question. So the risks with this particular work were relatively low and are sort of the right model. If you will, we try to reduce the amount of information that we leave on these servers, right? We assume that any point in time that you get flagged is being owned or controlled by us and just shut down and we lose access to whatever was on them. So we are very careful in how we access those machines, making sure that all the data that we're collecting is removed quickly so that we don't, of course, have setbacks, as well as reducing what may be accessible to an adversarial nation, like China, were they to do conduct fair-insics on one of these servers?
But aside from the sort of technical risks there, there's also kind of these risks to the people doing the research, right? In this particular work, again, there wasn't much of a concern. We were just doing this sort of observation of what was occurring. But in some of our other work, we have really struggled with these concerns and even struggled to have people accept the research that we have done due to the legal risks associated with it. Some people refer to some of our work as being an offensive attack against these services that are deployed in China.
Which is very interesting because, at least from what you described so far, you're simply allowing traffic to leave China and go, in this case, into the US. There's no sort of threat data there. It's not like you're doing a kind of reverse pen test or something. It's just information. Exactly. You're exactly right, in this case.
And now some of our other research, one particular attack, we found a memory leak in their DNS censorship that allowed us to leak vast amounts of memory off of these servers. That was when it would be a tax that we were maybe a little more concerned about, but it sort of goes to show how just observing this traffic or sending traffic back and forth isn't something to be as concerned about. Yeah. So you mentioned pop-down. That is an interesting concept that as you've just touched on, it refers to sort of the density or the bit density in the payloads.
But let's talk a bit more about that because it seems like that was such a critical kind of heuristic for how the DFW now operates. Yeah, absolutely. So pop-count is kind of a technical term or implementation of handling weight. Right. So with handling weight, we're concerned with like the number of non-zero symbols in some type of data.
And with pop-count, we are just counting the number of non-zero bits in some form of data. Right. So for each byte, we simply count, okay, how many bits have been set? And eight bits in a byte. So if we see four of those bits set, we can assume this is relatively high entropy.
Right. The closer we are to 50%. The higher likelihood that this is a packet containing high entropy and likely encrypted. Data will cluster around that number, right? Because if we have to flip a coin eight times, right, we would expect four heads and four tails.
Same thing applies when setting bits in a byte. And ultimately, this sort of speaks to how the GFW wants to execute this censorship. Right. They are doing the best they can to come up with crude but effective techniques. Right.
So this is a relatively simple thing to do, right? We're just counting bits. But that's the sort of technique that they have to deploy in order to stay relatively optimized and keep up with the vast amounts of traffic that they are constantly ingesting. Okay. And just to back up for a second, the sort of overall thing that was kind of discovered here was the idea that rather than selectively blocking specific things, it's kind of they'll be around now, which is everything is blocked.
Unless I guess in this case, like the bit density can be determined to be, well, I'm looking at the notes here, which is it has to fall within a very specific range. Is that right? For it to be. Yes. So it's funny.
The range was exactly 3.4 to 4.6. Otherwise, the count within that threshold, they would consider that traffic to be high entropy and they would ultimately block it. How they came up with this number? We're not exactly sure. Right.
That was just in our test. That's what we observed is the exact threshold for which they would consider it to be high entropy. Right. And so then for the GFW to sort of accept that this data is random and should pass effectively that as you touched on the clusters around like the 4.0 of it density. Exactly.
Exactly. So count all the bits that you have in the packet and see how many are set to one. And then so divide the numbers set to one by the total number of bits. Right. And if that value shows up between 3.4 and 4.6, then you can say, OK, this is clearly an encrypted payload and we're going to block it.
Otherwise, we'll have it. Gotcha. Yeah. So then there are kind of like other, I guess, exemption rules that I think you discovered as well around ASCII and vertical fingerprinting. So maybe we could just like dive into those as well.
Absolutely. And so I think what's important to understand here is that that entropy count actually came last. Right. So we refer to our rules as exemptions. Right.
So we're trying to find a way to exempt traffic from blocking. Sort of like you said, block everything and decide what not to block. And when we think about pop count, right, we have to look at every single bit in that payload in order to conduct that entropy test. Right. And that's that can be considerably resource intensive when you're considering the scale of traffic they're observing.
So they tried to identify other tests that they could run first to sort of filter off that traffic. And there were two different ways that they did this. The first thing that they would look for large amounts of ASCII. So if they saw a continuous string of ASCII characters and code, ASCII characters, they would exempt that traffic if it started with ASCII characters, they would exempt that traffic. And that was effectively just saying, OK, this is unlikely to be encrypted data.
Right. So we can even though we don't know what this protocol is, we can allow it. Right. The other approach they took to filter off traffic before they reached that entropy measurement was through protocol fingerprinting. It was really a crude protocol fingerprint.
But effectively, they chose a handful of protocols that they knew they saw a lot of, for example, TLS looked at what the first few bytes of a TLS connection were. And then if they saw those bytes at the start of any packet, they would say this isn't going to be considered a blank encrypted packet. Right. And that did two things. In the case of TLS, you're able to filter off the vast majority of your traffic and the measurements that we conduct here in that regard, it would be about 80% of traffic will get exempt right there with looking for TLS fingerprints.
But in addition to that, if you think about a TLS packet, right, we've got a couple of header bytes at the top and then we have an encrypted payload. So if you aren't filtering off that traffic, you will have an insanely high false positive rate because so much of the internet is encrypted. It's just encrypted in a way that we're OK with, right, as long as it's TLS, that's not a problem or as long as it's SSH, that's not a problem. Yeah, I wanted to then touch on false positive effectively, sort of you call it sort of collateral damage to like legitimate traffic. If you like how I think there's a lot of even the resources estimated around 0.6% of like the normal internet traffic would be false positives.
How did you, I guess, arrive at validation? Absolutely. So the false positive rate, we concluded was 0.6% exactly right. And the way that we did that is by taking these rules that we had determined and testing them against the nine traffic that we had here at the University of Colorado. So we are fortunate to have a resource here that allows us to see all of the network traffic going to and from the University and we rolled out these rules and basically looked at the traffic that wouldn't be blocked or that would be blocked if it was conducted here at CU.
And what that allowed us to do is say, OK, knowing that the University of the United States is going to be using censorship, circumvention tools or proxies, right? Because there's no blocking occurring. Why would they go to the trouble? And so we could sort of infer that this may be the false positive rate and that allowed us to identify a couple of other exemptions that they were doing. We took some of the bytes that we observed in those packets that would have gotten blocked here at CU and we found some other TLS resumption headers that they were allowed to sing.
And we found the large majority of the traffic that would have gotten blocked here at CU ultimately belonged to torrent services, those specific protocols for torrenting. And so we believe that there may have been some reason that they still wind up blocked that traffic anyway. So their false positive rate might have been seen as even lower based off of that. Yeah, very interesting. I guess that's sort of an interesting piece of doing it within a university.
There's obviously a lot of traffic generally. So yeah, I mean, was there any sort of like, if you were to do this again, for example, would there be any benefit to doing it in a sort of more siloed environment, if you like, or do you think that would not really affect the results? Do you mean for calculating the false positive rate? Yeah, exactly. Well, I don't think so.
And the reason being is that having access to this sort of network that has just a large amount of random traffic should be in theory representative of the type of traffic that would be coming and going across that national border. Right. And so our hope is that we're modeling that as close as possible. Yeah, that's a really good point. OK.
So if we look at the idea of podcast manipulation, so I think I'm going to let you take us through this. So I mean, you are a programmer and I believe a lot of this was done in Rust. So we've got to have a lot of Rust listeners as well on the podcast. So could you just take us through of what even is what kind of manipulation and why was that part of the research? Absolutely.
So once we had sort of determined, OK, this is how they're doing this blocking, right? What can we do to help proxy developers get around this blocking? Right? This is one of the most common techniques for censorship, so I can mention we've got to get it working again. And our solution was we knew that they didn't have some fancy metric for determining whether or not it was encrypted.
Right? It was just this pop count. So what if we stuff the payload full of additional bits, right? We can check the pop count of our own packet that's about to go across the network. And if we see that payload is highly and tropic, which, of course, it's going to be because it's encrypted, we can instead add some ones or some zeros, depending on which way we need to go and get outside of that encrypted threshold range.
Right. And so that was one of the more sophisticated techniques that we deployed in order to stop this censorship from being effective was we would add bits to the payload using a scheme based off of the key for the encryption so that it was pseudo random, right? There wasn't just a bunch of ones at the beginning or a bunch of zeros at the beginning or something. And then we would add a few bytes at the end that would tell you exactly how many bits you need to remove, right? And so then when the server or the other host receives this packet, they can decrypt how many bits they need to remove, and then they can determine which bits are supposed to be removed based off of that key.
And ultimately, this got implemented in shadow socks's rust implementation, as well as the shadow socks Android implementation. And it's been working to this day, I believe. Well, and like what kind of performance hit or overhead would you say this ads, if any? Sure. So are we estimate that the overhead is about 17% so if you assume a worst case scenario, if you have a pop count of exactly four, right?
Meaning we have four zeros, four ones, and every single byte on average. What we need to do is add just enough bits, in this case, we'd be adding ones, right, to exceed the pop count of 4.6. And what that works out to is about 17.6% overhead, which we find to be very tolerable, right? Given the overhead is already required for doing this sort of multi-layered proxy, we believe that a 17% overhead is something that you can apply to every single packet, and not necessarily just the first few packets of the connection already good. So is this like shadow socks rust to your knowledge?
Has this been used within VPNs, like commercially available VPNs? Or like, how has this sort of been taken up beyond the sort of research stage? Yeah. So shadow socks Android isn't that you can download, right? And you can use as a client, right?
Shadow socks rust is a, they provide like a command line implementation, but it's also designed as a library that can be added into other VPNs. Off the top of my head, I don't know the names of like the commercial ones that do use it, but it is heavily used by more user friendly applications. Yeah, this is something I'm probably, I'm going to go and do some extra research on, you know, where certain, I wouldn't say the name names, but like certain VPNs have like said, oh, we've got this like new protocol and it's faster, better, can get through anything else that might not be working. So I'm curious if maybe this is what actually we've been referring to building into their commercial products. So that's a little note for me to go on afterwards.
I think you also came up with other sort of circumvention methods. One is sounds very simple, but I mean, it must be a little more to it, like adding an HTTP or a TS header. How does that even work? You're exactly right. It is as simple as it sounds.
And so we found that there were these exemption, basically byte strings, right? That would be the first four bytes of a TLS connection, right? And so we said, OK, can we just add those four bytes to the start of a fully encrypted, fully ran and tailored? And the answer was yes. And we were like, well, that was really simple.
And so what we ultimately ended up doing is we provided this information immediately to the proxy developers, right? We said, hey, guys, we're working on this. We're still not exactly sure how it works. But we do know that if you add these couple of bytes to the start of every payload, your chat will get through no problem. And so we sort of use this as a emergency solution, right?
A little bandaid to give out that would allow people to get their services back up and running while we developed the kind of more sophisticated approach behind the pop-down manipulation. I mean, is there any evidence that the GFW is like actively probing suspected proxies like this or? Absolutely. So active probing was a technique that was around long before this particular incident. And it's one of the primary concerns of proxy servers, right?
Is that the GFW will see a connection to an IP address, right? And they'll be like, that connection looks a little weird. We think it might be this proxy implementation, but we don't want it to go block an IP address really, really. So what we're going to do is we're going to make a couple of connections to that IP address and see if we can get to talk to us in a way that is representative of one of these proxies, right? And so this is a common technique that they were using.
And it was actually they were doing active probing with sort of this entropy check prior to the passive detection of what is going on in this paper. And I do think that that's an important distinction to make is that this was a passive attack where they were just blocking any traffic that met these parameters and it wasn't affecting specific IPs. There were some that were specifically targeted, but it wasn't like this specific IP is a proxy and we're not going to allow traffic to it. And like, I mean, is there any sort of defense against this or is just kind of part of the landscape and it's going to be for anyone that's like trying to do this is just sort of a cat-n-mouse thing? Well, and cat-n-mouse is the term that we frequently use, right?
Is how are we going to get just one step ahead this week? And there are a couple of different defenses or strategies that we can use to prevent active probing. There are techniques like trying to behave the same for every single connection, right? So you might try to have a standard web server, right? And it will receive standard TLS connections, standard HTTP requests, right?
It will respond to those. But when it gets special connections, right? Proxy connections, it will handle those differently. And the hope is is that by using some type of cryptographic key that's sent by the client, the sensor will never be able to determine that it is in fact a proxy, right? They will have access to that key in order to get that host to respond in a specific way.
There are other approaches, kind of similar vein in which use some sort of other application as a front, right? So the naive proxy is known to basically use Chromium as its sort of front application and then the proxy sort of runs on top of Chromium, right? So everything looks like it's a standard Google Chrome TLS connection, but in reality it's a tunnel for naive proxy. The third approach that we also use here and has been a large chunk of our research is refraction networking. And in this case, what we do is we say, okay, you can send a connection to any IP in this range that we can observe, right?
Say like the University of Colorado's IP range. And using this network tab that we have access to, we'll just look for connections that look like they belong to us. And when we see one, we'll pick that connection up. We will start our proxy tunnel with that connection and redirect all that traffic to its inton destination, right? And why that's so advantageous and sort of defeats the cat and mouse is now the sensor has to decide, okay, in order to block this proxy, we have to block every single IP address associated with this American university.
And so we really raise the stakes of the false positives associated with blocking that sort of traffic. Interesting. Okay. So we look at deployment, like once this was kind of all discovered, if you like, did you have to do like responsible disclosure and like, who do you even make that to? And how does that then I guess filter through into sort of actual non academic users, I guess?
Yeah, sure. So responsible disclosure is a funny subject to bring up when you're when you're attacking a system like the GFW, right? They're kind of two sides of that coin, right? We are disclosing our findings to the circumvention tool developers. But then there's also this idea of, well, do we disclose something to the GFW?
Like, is that something that needs to happen? And in this particular case, no, right, we were just reverse engineering this technique that they had. And it's not usually something that we're particularly interested in doing, it's helping them figure out what we've learned. But we do work closely with the circumvention tool developers, right? So like I said, in regards to the sort of pending bytes, it seems very simple.
That was information that went out very early in this process, like January of 2022 is when we ultimately sort of found that and started giving that information out to these developers, so that you could start patching their tools as early as possible. And that sort of gets into what you were asking about how do we filter this out of the academic world and into the hands of people on the ground. And it's those same relationships, right, where we already work closely with the developers of these tools. And when we find something out, we can let them know before it goes public and they can try and get their users as quickly as possible. Yeah, I do find the whole responsible disclosure when then thinking, would you sort of disclose this to what the CCP, it's a strange one to think about.
That said, do you have any understanding that this research has been taken in on that side? And any lines of communication around it or sort of implicit or explicit changes that you've seen as a result? So with this particular research now, there wasn't really anything we told them, right? And there wasn't really anything that we could observe. We'll get into some more of the time, I think, in a little bit.
But on other works, like I mentioned, we had these offensive attack in their DNS injector. And we did try to disclose that to them. We sent emails to C insert. That was who we decided was the best plan of contact. We never heard back.
But what we did find is that there was patching of this vulnerability after we had made this disclosure, right? So we do believe that they, to some extent, are listening and some extent are paying attention to what we're doing. But it's still that black box where you don't have 100% confirmation. And yeah, as you alluded to, there's sort of another timeline marker here, which is, I think it's March 2023, that the GFW actually stops dynamic blocking. Could you talk to us about what you understand from that?
Yeah. And again, this is one of those parts where we can speculate a lot, and I can give you some of my thoughts on why. But the facts are we don't know, right? What we do know is that the blocking began on November 6 of 2021 and ended on March 15th to 2023. In that timeframe, there were a number of political things going on in China.
Xi Jinping was running for re-election in the beginning of March of 2023. I believe he was like re-elected two or three days before the blocking stopped. And we've seen that pattern occur, not just in China, but in a number of other countries, where when politically sensitive events are occurring, they will sort of ramp up their censorship. And then when those events sort of settle down, they tend to turn off their censorship or ramp it back down. What's interesting with China is usually when they find a technique that they like, they stick with it, they keep it going kind of ongoing.
And so what was different about this technique is unclear. We continue to do longitudinal measurements to see if this sensor has been turned back on. We still haven't seen any evidence that has been re-enabled. But we can only speculate that potentially it was too computationally intensive, or that the vantage points required were too valuable to be using for this technique. Yeah, and I'm thinking of the dates again, 2021 and 2023.
I mean, do you think the intertwines with politics as you've touched on? But the pandemic, that was also quite a sensitive time for China from an information standpoint, because it seemed like at least I was living in Hong Kong at the time, which had people often refer to Hong Kong in the lockdowns. And I can say categorically, there were no lockdowns in Hong Kong, but there were for sure lockdowns in China as a reference, being able to leave your house and so forth. And I think it was quite sensitive as to people within China being able to see what the rest of the world was doing, especially towards maybe towards the end of the rest of the world, because the end of the pandemic seemed to be the worst part for China. And as you say, it was ramping up into a re-election, if you want to call it that off-shooting thing.
So anything there that you think could be related as well? I think it's very possible. Again, that's one of the things that makes this research the most challenging, though, is you never really know. You don't get copies of the memos sent around the government that say, hey, turn on this censorship on this day for this reason. And so you're exactly right.
It could have been certain events going on, like sort of the tail end of the pandemic that was causing the censorship to ramp up. It could have been the political timing. It's really hard to say. Yeah, China in a nutshell. Exactly.
Yeah. And like today, here we are in 2025. And just like, what is the current status the same effectively that is off or have you seen anything change? Yeah, in regards to this specific technique, we haven't seen any activity. That's not to say that the GFW is turned off.
This is one small piece of the GFW. And they're still deploying a number of other techniques. But the idea of passively observing and blocking fully encrypted traffic, we are not seeing any signs of today. Looking at just speculation on how this was actually operating, even though, as you've just touched on it, this specific part is probably turned off about like general GFW architecture and just speculation on how that all works. Like, what do you think it is?
Yeah. So from a technical perspective, there are a couple of different vantage points that they could have access to, right? And the two that sort of apply here is what we refer to as like an empath sensor or vantage point versus an on path vantage point, right? And an in path vantage point sits quite literally in the path. It is one of the hops that your traffic takes between the two hosts, right?
And what's special about that vantage point is that it has the ability to tamper with that connection, right? It could change the payload or it could go so far as to just start dropping the packets. And because of the blocking technique that we observed in this research, we believe that they were using these empath vantage points because they were just dropping the packets afterwards. The other side is what's called an on path vantage point, right? This is a less sophisticated or less valuable vantage point where they get a copy of all of your traffic and they can even send packets back to the hosts, but they aren't sitting in the path where they can manipulate and potentially drop that traffic, right?
And it's an important distinction because those empath vantage points, if something goes wrong, right? That could lead to inner nowages, right? And you could have major issues there. So I believe that those are potentially sensitive assets and they are very careful of what they allow run on those. Most of the censorship that we observe ongoing tends to be more of an on path attack.
And do you think there's any ML machine learning kind of going on there or is kind of more of a static, I guess, implementation? Yeah, so I think that for specifically what we observed, they had to have deterministic code that was running, right? They had if statements that were checking those bytes, right? And checking that entropy because, you know, ML inference is just too slow to be doing in line, right? However, it could be possible that they were using ML to try and determine some of those heuristics, right?
It's hard to say. Though we have confirmed that the GFW is using machine learning in other aspects. There's been a massive leak of information in the past couple of months coming out of a Chinese corporation called GEDGE and a research lab called MESA. And we have found evidence in those repositories that they have been using machine learning to classify data and determine what traffic is coming from, they sense or not. However, I think that most of that research is being used in those active attacks, right?
Those attacks where they're making the connections because to try and do it passively at line rate is just unrealistic. Gotcha. So as something that kind of comes up in my line of work is, again, I sit in Singapore and so I'm working currently with products that do get used within China. And people sometimes come to me and say, like, oh, do you think there's going to be a problem with this team using, when I say problem, I mean, like a performance or just access problem with this team using the product within China, I give a very kind of this point, lame answer. The last time I went into mainland China is probably 10 years ago at this point.
I mean, so quite often than just stopped. But my understanding is always just that there is this massive speed, the massive latency around traffic that goes in and out of the perimeter, if you want to call that. And that's usually my kind of standard piece of advice is just if this is not something that has infra based within the GFW. So that's usually why people strike up deals. I believe Cloudflare has like a product that's kind of like a deal where they've had to, you know, JV joint venture with a Chinese partner so that that's a way to enable your product to get over this latency.
I think you sent me a little piece on this before we chat today called and it's like with the great bottleneck. So yeah, maybe could you just talk to us about that? Because I think I would also love to like learn a little bit more around what actually is this rather than me just saying, oh, this is just like a latency problem. And that's what you should assume. Sure.
Yeah, we've got to get more creative with our naming. I mean, the GBE something. Yeah, exactly. It's always got to be the great. Yeah.
But yeah, so this has been an issue for it seems like a long time. It's an issue that affects, you know, more people than just the citizens of China, right, who are trying to access things they aren't supposed to, right? The great bottleneck is this sort of widespread idea that traffic going outside of China is going to be slow or it's going to have some type of network issue, right? And the research that you're referring to, I think, found that was primarily on the download side, right? So when we are pulling traffic from other places, we have an issue or pulling data from other outside of the country and we have an issue.
The upload doesn't seem to be as big of an issue. I think that this primarily isn't necessarily a censorship technique, right? I think that if it was a result of the censorship that we've observed, if they didn't have the compute resources to effectively censor this traffic, we would see it more on that upload side, right? When that HTTP request gets made, that's when you would see the slowdown, because that's the traffic that you are trying to observe for a potential censorship things. And so I think it is truly just a lack of international infrastructure in China.
I think that, and I think the reason that they're doing that is because they sort of have this goal of creating this isolated ecosystem, right? They want for their users or their citizens to primarily be accessing Chinese-based sources, right? Chinese tools, Chinese services, what have you. And so I think that even though they could probably afford to spend the money to improve that international infrastructure, build some more cables under our C cables, what have you, they are choosing not to almost as a way to incentivize both companies and users to use national services. And so yeah, I don't think it's necessarily malicious, but it's more pushing towards this goal of isolation.
Yeah, the cloud providers are sort of another proxy battle, I guess, where you've got like Alibaba Cloud and others within China. And then obviously the big three that we know in the US and the rest of the world. So I think that's an interesting take on it, which is, yeah, this probably has like as much of a commercial leaning as it does. I think that's nothing where people go into China and I asked a friend who goes more from the idea, I said, oh my family member, going to China, like what should I advise them? And he just said, oh, you must download this specific app.
Yes, it's half in Chinese, but like this is how you use stuff, you know, like call a cat app or I mean, that's like maybe seems obviously you go to another country and okay, like in Singapore, say, hey, get the grab app, like, because we were just not thinking here, but this is like a whole layer extra, which is sort of like you're going to be able to achieve nothing unless you actually download this super app and has your maps and transport and food and all this kind of stuff. Everything else just isn't going to help you out, basically. I think you're exactly right. This isn't something that's unique to China. I mean, that sort of super app seems an extreme example, but just like you said, there tend to be ride sharing apps that exist in certain regional areas, or even like the US trying to block TikTok unless it gets sold to an American entity is really the same thing, right?
It's this kind of idea of we are going to prioritize national services before international services. Yeah, absolutely. And yeah, I mean, I just carve out with this, is that Hong Kong still doesn't really experience this bottleneck effect? At least that was my when I was last year, there is no sort of GFW per say within Hong Kong and is it used as any kind of like proxy or something? I think we were touching this before we started recording.
Yeah. So there is sort of this idea that Hong Kong tends to have a freer internet in our perspective of things. I would definitely defer to you on any lived experience, but we do see that number of proxies will kind of use that as their first hop. And there's even some speculation on if there are gray market links that exist between China and Hong Kong, specifically for the transit of proxy traffic that's ultimately going to go other places. Some people even believe that a lot of the proxy traffic exists simply as a way to get around the bottleneck, right?
They might not be trying to access undesirable content. They just have some type of maybe business need to get outside of the country. They aren't like at the enterprise level where they can afford these contracts with the cloud flare and so they use these proxies as an alternative means. Yeah. And this crosses into like the realms of like dark fiber, I think.
Yeah. Yeah. So I mean, my kind of anecdotally, this is just like you, I can be sitting in a Starbucks here and I'm over here in conversation of someone who's could he come from Hong Kong to sell a dark fiber contract to someone in Singapore. Could you maybe just touch on that briefly? Like how does that cross into this?
I'm not much of an expert on that. I've only heard sort of like you said, these anecdotal experiences of somebody has access to some type of link that goes from mainland China to Hong Kong. And then they're selling access to that to people in China, but I'd love to know more. This looks, I mean, at least to me, it kind of looks legit. You know, this is but equally, I don't know, why is it being held in a Starbucks and not in someone's office?
That's maybe that's all we need to know. If we sort of then just look at any other countries that you at least believe may have adopted these techniques, I mean, maybe some obvious ones that people are already thinking of, like just from where you sit, like which other places have you maybe seen adopt the GFW approach? Yeah. So there are a couple of different things to consider there. Specifically for the GFW approach, I think it's interesting and pertinent to get back to that sort of leak that I was referring to.
And you may have heard about this. It sort of hits on the main stream news sources and that this massive trove of data got out of this Chinese corporation called GEDGE. And this corporation's goal is effectively to commercialize censorship, software and hardware, and then distribute it to other nations. And in that leak, they found that Kazakhstan, Ethiopia, Pakistan and Myanmar are all confirmed clients or customers of this organization that is GEDGE. And so we know that they to some extent have the capabilities of the GFW through these business dealings.
However, those aren't the countries that we typically think of, right? When we think about our nations that are doing censorship. I think one of the big ones that comes to mind is Iran, right? We don't know if there's any relationship between Iran and China's censorship infrastructure. Iran, they all tend to have their own sort of character, if you will.
Iran is known to be very dynamic. They are constantly changing what's getting censored, how it's getting censored, where it's getting censored. And two neighboring ISPs in Iran can have completely different censorship experiences. And then another one is Russia. I mean, Russia, again, has sort of its own interesting culture where they have more of a capitalist spin, if you will, on their infrastructure, right?
In China, you kind of have two or three major ISPs. They tend to have heavy involvement from the government. And so deploying that censorship infrastructure is very much a state activity. But when you look at Russia, the censorship is basically the state saying to all of these sort of independent ISPs, Hey, look, you're going to block this list of sites. We don't care how you do it.
Figured out, right? And so Russia has its own way of determining or you see these weird characteristics where one ISP will look different from another and they seem to be doing almost this bare minimum approach, right? Where it's very easy to get around and it's very patchwork. So yeah, the censorship landscape is fascinating. How much of a shift do you see between all of these different countries?
Yeah, for sure. And I guess just looking ahead, if you want to call it that any predictions where things go, I mean, we've touched on like a lot of the whole idea here is that they just seem to keep changing and there might not be any rhyme or reason, at least from the outside as to why. But yeah, just if you were to kind of think ahead, anything that you suspect might change or happen. It's hard to say it is a dynamic landscape. But I do believe that censorship is going to become a little easier for the sensors.
And it's going to become harder to circumvent. We're seeing, especially in China, this shift of middle boxes closer and closer to the end user, where due to sort of a scarcity of IP addresses, they're implementing these NATs to where a local residential address won't have a dedicated IP. Right. And so as we sort of see more and more middle box infrastructure getting closer and closer to that end user, it will become easier for them to deploy more of a distributed censorship network that can kind of implement more advanced techniques because it's taking on less load. So just as we sort of start to wrap up here, what about your own research?
Are you doing more into this kind of going forwards? Or is this, are you kind of leaving this here and moving onto a different part of the censorship landscape? Or yeah, where are you taking things from here? Yeah. So I've been looking extensively into that leaked information I referred to that's a relatively new finding for us.
And so we've been looking at that to sort of be that ground truth that doesn't exist, right? Confirm some of these statements that we have made and all of the research we've done so far, as well as trying to understand better what role machine learning can play in this space. Right. So a lot of my research is focused around that network tap here at CU and having access to this massive corpus of internet traffic that is real and finding ways that we can better classify that traffic or even generate synthetic traffic is sort of the direction I'm headed in next. Awesome.
Well, that all sounds fascinating. I hope maybe we get to catch up in a couple of years here. What you've been up to since then. And just want to really thank you for your time Jackson. And I've learned a ton today.
I'm sure the audience has as well. So obviously you're doing great work. It's very important that this information is understood at least outside of China. So yeah, thanks so much. Absolutely.
Thank you so much for having me.