AI against Censorship: Genetic Algorithms, The Geneva Project, ML in Security, and more!
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper security machine learning in security ai security ai network security deep learning censorship ai censorship internet censorship geneva vpn genetic algorithms genetic algorithm genetic algorithm example real world genetic algorithm ai in the real world firewall evolution evolutionary search maryland breakerspace encryption amplification
#security #censorship #ai

Most of us conceive the internet as a free and open space where we are able to send traffic between any two nodes, but for large parts of the world this is not the case. Entire nations have large machinery in place to survey all internet traffic and automated procedures to block any undesirable connections. Evading such censorship has been largely a cat-and-mouse game between security researchers and government actors. A new system, called Geneva, uses a Genetic Algorithm in combination with Evolutionary Search in order to dynamically evade such censorship and adjust itself in real-time to any potential response by its adversaries. In this video, I talk to Security researcher Kevin Bock, who is one of Geneva's main contributors and member of the Breakerspace project. We talk about the evolution of internet censorship, how to evade it, how to mess with the censors' infrastructure, as well as the broader emerging connections between AI and Security.

OUTLINE:
0:00 - Intro
3:30 - What is automated censorship in networks?
7:20 - The evolution of censorship vs evasion
12:40 - Why do we need a dynamic, evolving system?
16:30 - The building blocks of Geneva
23:15 - Introducing evolution
28:30 - What's the censors' response?
31:45 - How was Geneva's media reception?
33:15 - Where do we go from here?
37:30 - Can we deliberately attack the censors?
47:00 - On responsible disclosure
49:40 - Breakerspace: Security research for undergrads
50:40 - How often do you get into trouble?
52:10 - How can I get started in security?

Learn more at:
- Geneva (& more) project page: https://censorship.ai
- Open Observatory of Network Interference: https://ooni.org
- Censored Planet: https://censoredplanet.org
- Breakerspace: https://breakerspace.cs.umd.edu


Links:
Merch: http://store.ykilcher.com
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hello there, today I'm talking to Kevin Bok, who is a cybersecurity expert and one of the main people involved in the Geneva project. Geneva is a genetic algorithm that evades censorship by nation states. So in real time, Geneva can evolve to the ever more present danger of censorship by really big entities such as governments. All of this is done through an evolutionary search over a program grammar. And in this interview, we're going to touch on a whole range of topics including Geneva, how it works, what it does, why people research it and what it has done so far in the world, but also the broader topics of security and its connections to AI, how people can get started in this field and what the main questions and problems are in this space. Further, Geneva comes out of a project at the University of Maryland called Breaker Space, which is a sort of lab that includes undergraduates in security research, which is a really cool project. And I think highlighting this would be helpful to some people. Maybe you're at the university, you don't know this exists. Go there, take part. All right, without further ado, I want to give over to the interview and have fun. All right, everyone, I have with me today here Kevin Bok, who is a PhD student at the University of Maryland, a cybersecurity researcher, and a member of Breaker Space, which is a pretty cool project at the University of Maryland. He also has been in the news a little bit with a project that's called Geneva, which uses genetic algorithms to evade censorship by nation states. And I think that's pretty cool. So Kevin, welcome to the show and thanks for being here. Thank you for having me. I'm excited to be here. So the goal of today, it's a little bit different because I'm a total noob at security. Most of the audience of this channel is into machine learning. Maybe some know about security, some know about the censorship apparatus that's in place around the world and what people do about it. I think most won't. So today I'll be asking mostly noobish questions and we'll have you here to guide us through everything, to guide us through what's happening in this world. So maybe you first can start off a little bit. How did you get into, how did you get to the place where you are? What's the main things in security right now that draw you to it? I think security and the censorship space also is in this really cool time where AI and ML techniques have been exploding in all these other fields and they're just over the last four years really breaking into security and we're still figuring out all the different applications where you can apply these techniques in security. There's new techniques and new applications that people are discovering all the time from better ways to detect spam and better ways to identify, hey, this domain is malicious or AI-based scanners for that binary you downloaded, that's probably malware, things like that. So security field is still discovering all sorts of new ways you can apply these techniques and that was one of my motivations initially actually of bringing this to censorship because this project was really the entire field of censorship's first foray into using AI and ML-like techniques. And if you talk about censorship, what do you mean exactly by that? Yes, there's so many forms of censorship in effect around the world today. I mean everything from political pressure to self-censorship to taking down... Like there's so many different types. So I'm going to scope this discussion down a little bit, just the type of censorship that we study in this lab and that's this type of automated censorship that happens in the network performed by nation states. So what do I mean by this? If you're a user in certain regimes around the world, let's say in Iran or something and you try and make a request, as that request, as that web traffic crosses through the border of the country, it is scanned, parsed and inspected by some machines that physically reside in the network called middle boxes, because they're in the middle of the network. And these middle boxes examine your request and they say, is this something we should allow or not? And if the answer is no, they either inject traffic to take down your connection or they drop your connection or they do something to disrupt what's going on. And you'll notice everything I just said there, there's no human in the loop. There's no human content review or anything like this. It's a purely automated run by these middle boxes or firewalls deployed by these nations that just automatically inspect the internet traffic as they go by. So that's really the scope of what we've been studying here. Naive question. Why can't I just encrypt my traffic and then every traffic looks the same towards the outside? Yeah, that's a great question. So why can't we just encrypt everything? People have been trying. So there's like a couple of different approaches to this. You're like, well, let's just use HTTPS, right? Encrypted. We're good. Unfortunately, HTTPS has a small privacy leakage. When you first set up an HTTPS connection and that very first initial is called a handshake and that first back and forth, you as the client, as a part of the protocol, you have to announce the domain you're talking to. And that announcement happens unencrypted. So if you're making a HTTPS handshake to Wikipedia, in the very first packet you send, it's going to include the word Wikipedia. And that's called the server name indication field. You indicate to the server what the name of the server you're trying to talk to. And unfortunately, sensors just read that fields and then they take down your connection if you talk to a forbidden domain. So HTTPS, unfortunately not close, but not quite finishing the job. Now, I will say there have been just a quick sidebar. There have been some advancements in HTTPS to try and fix this. There's a recent proposal to encrypt that fields. It's called encrypted SNI. And China just started censoring that last year. So you can try and encrypt things, but these sensors are often just hostile to the idea of just letting their citizens just encrypt all their traffic. I guess it's a little bit like if everyone encrypts, like with HTTPS nowadays, everyone does it. So you can't conceivably block HTTPS just because you don't like some traffic. But if there's a new type of encryption, it's probably only the people that have something to hide that use that type of encryption. So is a strategy that the rest of the world as fast as possible would use these techniques to kind of make that approach unusable? That's exactly right. The broader topic you're actually discovering and saying out loud here is this idea of collateral damage, of can we make a protocol or something so popular and use so diversely that if a sensor were to try and block it, it would cause irreparable harm to good services. There's some meaningful cost to performing that censorship. So just like you've identified HTTPS, that's everywhere. They can't just shut down all HTTPS. But rolling out a new encryption method for HTTPS that's not very widely deployed, they can nip that in the bud and prevent its rollout. So there's kind of this interesting race in a game between developers and these sensors that's still being played out. Now let's talk about more, let's say, naive approaches. What is the development of the field? What has been tried before and what has been, let's say, thwarted? Or what's the cat and mouse game looked like in the past? I imagine different things like there's Tor, there is all kinds of things. There is probably things that everyone installs on their end, like VPNs and tunnels and so on. What's been the general development over the years? Yeah, so the researchers and sensors have been playing this cat and mouse game for two decades now. And it's kind of evolved and it's been playing out in multiple fronts. So you're exactly right. Tor has been a huge front on that war, if you will. We've developed Tor and continue to advance it. Unfortunately, there are some limitations, just the Tor protocol and sensors can enumerate the Tor entry points basically and just block you. So once you get into Tor, you're generally great, but they try and block you out. There's been all sorts of techniques people have proposed, like maybe I can disguise my traffic to look like Skype. And then the sensor's like, well, you didn't disguise it quite well enough, blocked. There's a whole interesting field of defeating censorship or subfield, I should say, called packet manipulation based censorship. And this is this idea where all our communication is happening via packets. And if you just tweak those packets in just the right way, you could cause the sensor to miss you. And historically, that's also been something that's played out in this cat and mouse game where researchers will study these sensor systems and then they'll find a loophole and they'll deploy it and use it. And then the sensor's like, oh, I'll fix that. And then we're back to square zero. So this game has really been continuing to play. I'll call one thing out real quickly about VPNs. Because a lot of people, particularly those who have been to China, are like, I've been able to use a VPN and it's been OK. VPNs in many places work. In many places they don't. There's a country in the news recently. They were in the news because they rolled out a new law that forced their citizens to swear on the Quran that they would not use a VPN in order to get internet access installed in their homes. It's just like crazy sentence to say out loud. But in China, for example, these VPNs, many of them work most of the time. But what researchers have noticed is that around the time politically sensitive events are happening or political, such as elections, things like this, a lot of VPNs will just mysteriously stop working. And then after the event, they'll mysteriously start working again. And it kind of points to this broader idea that some of these countries may be sitting on more censorship capability than they deploy on a daily basis. And they have more power than they use. So this cat and mouse game may even be stronger than we think it is. Can you give us an idea of what this packet manipulation evasions look like? Because I imagine something you mentioned before, if there's Wikipedia in the header, I don't want my population to see Wikipedia. Like that's it. What can I possibly manipulate there in order to get through such censorship? Yeah. So we can think about sensors as our computers are sending packets around. You can imagine a lot of that communication like you're writing mail, your packets are envelopes that are going to the network. And in order to have a communication with a server like Wikipedia, that's going to take a couple of envelopes back and forth. And the sensor is just like the postman in the middle reading all your letters. And unfortunately that postman has got to process a lot of letters, a lot of letters. And you can imagine something the scale of like China, you're dealing with a huge, huge volume of traffic just at a constant basis. What that means is the sensor can't just remember everything it sees. So for example, if it's trying to track that, hey, that person over there is trying to talk to that server over there and that person over there is talking to that server over there, that state it has to maintain. And the amount of state it has to maintain, it'll grow. And the size of some work like China, it could grow pretty fast. So they have to be really careful about what they remember and the state they maintain. So you could imagine doing something like, let's say we're exchanging packets. There exists a type of packet called the reset packet. And these are normal packets our computers send these all the time. But they basically just exist to tell the other side, stop talking to me immediately. I'm hanging up the connection. So you can imagine doing something like you and I are communicating, we're sending these packets back and forth. And I just slip one additional packet into the connection towards the beginning and it's a reset packet. And I'll send that packet along. And when the postman sees that packet, he's like, well, these guys have stopped communicating after this message, he's going to ignore him forever. And then he throws away the state he's maintaining about our connection. He forgets that we're talking because why would he need to remember anymore? He thinks we're done. And if I craft that packet in such a way that it won't make it to you, or you'll see it and ignore it or something like this, then we'll be able to still communicate fine, right? Or our communication is unimpacted. But any of the packets that go by, the sensor's like, I don't know who this is. And you can get through. So this is like the broad strokes, this idea of packet manipulation based censorship, where you're tweaking the packets that go by to try and basically trick the sensor that's in the middle into letting you continue to talk. Now do I see this correctly, that there have been like a giant amount of these schemes proposed and as you say, there's a cat and mouse game. One is being proposed, then they fix it, then another one, then they fix it. So that points to the possibility of what if we could have something dynamic, right? What if we could have something that by itself tries to invent new things? And that's where you went with Geneva. Do I understand that correctly? That's exactly correct. Yeah, you're spot on. Yeah, so over the years, there's been, I want to say dozens of these that have been proposed and researchers have, it's exactly this cat and mouse game. They studied the censorship system. I mean, the censorship system is not public, so they're probing it, they're trying to take measurements. That's a lot of work. And then they get an understanding, they apply their good human intuition, they develop something cool and publish it and the sensor fixes it. They don't tell you they fixed it. They don't publish a paper that's like, hey, we just fixed your bug. So it just resets this to square zero. And so the idea with Geneva, which stands for genetic invasion, the idea of this was it's an algorithm that could kind of flip this process on its head. So instead of a human having to take the approach of let's understand how the censorship works and then defeat it, let's just have some AI or fuzzer or automated system, just attack the sensor, figure out ways through and then give it to the human. And now after the fact, my slow human brain can go figure out why that thing works. And now my brain is no longer the bottleneck to helping people get through the sensor. How does this, you want to go a bit more into detail? I mean, it sounds great at the surface, but there's a reason, right? We need security researchers probing, making sense. And there's a reason that's the bottleneck. If I were just to be like, well, you know, fuzz a bit, it's probably not going to work. So what does Geneva do that allows it to even be successful where maybe humans take a long time or wouldn't be successful? Yes, there were a couple of pretty significant challenges when we first started in applying something like a genetic algorithm or really any AI to the space of censorship. And if you think about the way censorship works, it's not hard to imagine like why that's the case. Because if you think about think about a censorship problem, right, like a query is either censored or it's not, it's just a binary decision. So it's not like your traditional ML or AI where you have this nice like gradient descent. There's no error. You're back from the sensor. The sensor doesn't tell you like, hey, if you tweak your query, just a little bit, you're getting closer. Yeah, you know, there's no gradient which with which you could work. So that that property alone rules out the majority of the ML field as far as approaches you can take. Is there even a loss? Like you said, it's hard to detect if you even get through. How do you do that in the first place? How do you notice success or failure? Yeah, so in our case, you're exactly right. Capture capturing that can be difficult. What we do to make it easier in ourselves is we obtain machines inside these censored countries and directly try to request for written content. So Geneva trains directly against the sensor and we know we got it. When the sensor takes action is kind of obvious. So Geneva will try and obtain some forbidden content while manipulating the packet stream. And then if it succeeds, great. If it fails, we'll know. Right. So this idea of how do we apply ML, AI, some fuzzing to this space? Like how do we build to this? There's a couple of main challenges towards doing that. The first is this total lack of gradient that I mentioned. And really that only leaves you with kind of a small number of approaches. And we chose to go down the route of let's use a genetic algorithm for this. There's some nice properties. It's easily explainable. You can understand how it works while it runs. It's a little less black boxy than something more like a neural net or something or Markov or something like this. But if you want to build a genetic algorithm, you need a couple of things. You're seeing what some of these strategies look like right here. So if you want to build a genetic algorithm, there's a couple of things you need. You need some building blocks. Something that the algorithm can compose and put together. And you need some way for it to put those things together. I mean, us humans as examples, as far as genetics goes, we've got our DNA bases, right, ACTG. And we can put those together in DNA. For the genetic algorithm for Geneva, we needed to decide what makes sense for building blocks for the algorithm to use. And that alone is like an initial really huge challenge because you could be creative and you can think about a million different ways an algorithm could manipulate a packet, right? Flip a bit. You could flip this bit. Like there's just so many different things you could give it to do. So one of the first challenges we had to figure out was how do we balance what this algorithm can and cannot do to the data it has? And on one hand, we could let it flip any bit. The downside of that is it could take forever to learn to check some, but it's super powerful. Like on the other extreme there, we could just encode what previous researchers found and let it play with those together. It would be super fast, but it'd be hard to learn anything new, right? We'd just be building in biases directly. So the approach we ended up taking was giving Geneva basically the same ability to change traffic as what the network itself could do. So the network itself has just a few set primitives that can do the packets. It can take a packet, make multiple packets, it can duplicate them, it can change a header to something, it's tampering a packet. You can take a packet, break it into multiple pieces, fragmenting. You can take a packet, drop it, which is just basically deleting the packet. So we built out these building blocks and then allow it to compose these things together in trees. So like syntax, you give it a syntax and it can assemble a little program out of this syntax, like one we see right here. That's exactly correct. Can you walk us through what this particular thing does? Sure, sure. This is kind of a fun strategy. So there's a few different components to a Geneva strategy. I'll break down the syntax for you real fast, what these programs look like. So the first component is the idea of a trigger. The trigger is what's between the square brackets. So there's two triggers in this, TCP flags S and TCP flags R. And when Geneva is monitoring traffic, the trigger tells it which packet should I act upon. So this first trigger you see here says TCP flags S. So that means that whatever actions are attached to that trigger will run on any SYN packet it sees. S stands for SYN. SYN means the start of my connection. So what this is going to do to that packet is the very first action we see is duplicate. So that means it's going to take that packet and make two of them. Now duplicate, the syntax of this is it's one set of actions, comma, another set of actions. So you'll see the two actions you see here are tamper and then send. So the second duplicate we do nothing to. So the second duplicate we're just going to send on the wire. But to the first duplicate what we're going to do is we're going to replace the flags fields in that packet with SYNAC SA. And then we're going to send that packet. So basically what this little program does is it sees outgoing SYNAC packets, outgoing SYN packets to your computer, and it duplicates them to make two packets and then replaces the flags in the first one with SYNAC. Now any networking person listening is like, this is clearly ridiculous. This never should work. Why would we even do this? Why are we talking about this? And what's going on here is that for certain sensors around the world, SYNAC is the packet that's typically sent by a server. It's never sent by a client. So what's going on in this strategy is when the client sends a SYNAC, the sensor says, whoa, I must have missed something. This client is clearly a server, which means the server must be the client. It reverses the roles of client and server in the mind of the sensor. And as a consequence, when the client makes the real request, since the sensor is processing packets differently between client and server, you're through. I see. So that's this idea of the strategy. So that connection in the mind of the sensor is already established as here's a server, here's a client, and it kind of keeps that state for subsequent packages. More or less. Yeah, that's exactly it. So this is an example of just one strategy in one of these programs that... So Geneva built this program itself and it built this through the process of evolution. And you've discovered, just to jump ahead a little bit because we're not through yet with explaining exactly how it works. But you've discovered that Geneva will actually reproduce a lot of the common or known or already discovered things that researchers have proposed, right? Yeah, we had this really cool result initially where we set out to try and... We wanted to, when we first developed this tool, kind of benchmark it against the rest of the fields. And that's kind of challenging because sensors have continued to evolve. So what we did was we sat down in the lab and we implemented in the lab our best guess as to what... Our best implementation, I should say, as to what these sensors looked like based on what previous researchers found. And then trained Geneva against these mock sensors and also trained it against the great firewall and real sensors where we could. And we found it was very quickly, it was able to reproduce basically the entire field. Every strategy a human had come up with, this also found and it found them pretty quickly. So it's really showing the power of automated approaches and AI ML. So you have... Let's get back a little bit. You have this syntax, right? That you can build trees from which are valid programs in Geneva. This will modify the traffic somehow. Now to say that most of this traffic will just not even be traffic probably, like the connection will be somehow bad. Some of it will go through and some of it will actually maybe evade the sensor. What do we need to get there? What do we need to get to a place where... I guess if you just do it naively and you randomize a little bit, it will just be bad. Like 99.9% of all the programs you generate, you'll initiate them and then after a while you'll see like my traffic isn't even getting anywhere, right? So what are the... Of the genetic algorithm components, what do we still need? Yeah. So we're building our way up to the genetic algorithm. We've got, just like you said, we got our building blocks. We got a way to put them together. We got a syntax so we can build these programs out of it. We can run these programs on network traffic. And you're exactly correct that if we initialize completely randomly, it's going to do terribly. And that's exactly what happens. We've tested this. So where do we need to go from here now that we have this? So this kind of brings us to this idea of let's get evolution in the mix. So you can imagine the way this works is we have a big pool of strategies. Okay, we'll call this a population. And each of these populations just take for granted for now that we have some diverse set of strategies in here. And we have a way to test them, right? We can try and make requests for something forbidden and we can run these programs on those requests as we make them. So for example, from inside of China, we can try and access Wikipedia. That's a sensitive resource. And we'll have these programs running on that connection. We'll just try and make that connection over and over again. And what we'll see is some of these strategies will destroy our connection. Some of them will just not work at all and do terribly. Some of them might keep our connection alive. And maybe if we get crazy lucky, we'll defeat censorship. But for now, let's just say a whole bunch of them will just destroy our connection and maybe some won't. We have is a fitness function. And this fitness function, this is a bar, a much broader space in ML and AI, but it's basically this idea of if you take some individual from the population, some individual strategy, how good is this thing? Survival of the fittest, like should this thing survive basically and continue to propagate its genetic material? So this was actually the second big challenge in applying AI and ML to this space of censorship vision of what on earth should a fitness function look like in this space? Because just like we talked about earlier, there's no gradient, right? And even come up with like a loss function can be a little tricky. And I mean, even if like, sorry to interrupt, but the fitness even like if the fit, I guess the fitness, is it anything else than zero? Like, okay, maybe some connections don't even work to like the server next to you. You can discard those. But other than that, the fitness is either doesn't reach the target or does reach the target. And if it does, you've kind of won, right? Like how can you even get a meaningful signal? Is there a fitness in between zero and one? Yeah, so and part of what makes Geneva work is we've kind of shoehorned our way to getting fitness between zero and one. And specifically what we do is rule out those strategies that break your own connection. So that's kind of how we've gotten between zero and one. Because it's not technically zero and one. It's almost negative one, zero, one. And negative one is Geneva shooting itself in the foot, right? It's just like dropping all your traffic. That's never going to work. And we shouldn't even bother exploring that space more, right? Like we're never going to go anywhere. But if you can make it so that your packets are at least interacting with the sensor and at least have the potential link to the server, well, now we might be getting somewhere. So basically what we do is we set up the fitness function in such a way that if strategies destroy the underlying connection, they'll be punished severely and basically killed off. And strategies that interact with the sensor, even though they get censored, they'll get a slightly higher fitness function than those other ones. So what's going to happen is because those individuals aren't, they're not successful, but they're still the most successful in the population pool, which means some subset of them will continue to reproduce. And basically that subset is just chosen randomly. But because we're just choosing randomly, mutation is still going to happen. So we're basically taking a set of individuals, they all interact with the sensor, and then we just mutate them and try again, and then mutate them and try again. And effectively what this has turned into is a fuzzer. Like Geneva is, the fitness function basically makes this a targeted fuzzer where we can fuzz just the space of strategies, just the space of programs that allow us to interact with the sensor. And then where it gets interesting is as this fuzzer is running generation after generation, just trying different crazy things against the sensor, if it finds something that gets through, suddenly that fitness is way higher than everything else. And that individual will start sharing its genetic material and propagating within the population pool. At that point, we could stop. We could stop the fitness function right there. But we optionally add some additional punishments and rewards for the algorithm at this point. And specifically we add basically a punishment for strategy complexity. So if an individual is successful, we optionally punish it for basically the number of actions and the amount of overhead it adds to the connection. And the reason we do that is this is not strictly required, but I have a very small, smooth human brain and it's so much easier to understand a strategy that's only two actions long, compared to some that's 50 actions long, for example. So if we could encourage the algorithm to be like, great, you got a solution, now simplify it down for me. And it will over the course of generations whittle it down to its smallest form and then at the end present to you its population pool and its best individuals. And we see here a few ways you can mutate. I think this just essentially comes down to changing the syntax tree in some form. Yep. And you can imagine all the different ways you can take these programs and mix them around. If you can think about it, Geneva can probably do it. And so just maybe for my understanding, but you're trying all of this, you say you have some machines inside of these countries. And I read some like, obviously this is not going to work against IP blocking. How do you not get IP blocked by them? I imagine there's some weird traffic that hits my censorship wall all the time. Why don't I just be like, well, gone. Yeah, that's a good question. And we get this question a lot, actually. And you're pointing to this broader question of what's the censor's response? You're doing all these wacky, crazy, ridiculous things. There's a strategy in there that just lights up every TCP flag. That package shouldn't exist flatly. It has no meaning on the network. But Geneva tried it, found it, and found that it works. So where do censors go from here? It sounds like, when we're talking about things like it's sending crazy packets, it sounds like that should be something that's easy to detect on the network. But it sounds easy until you try and write it. Because if you think about it, writing something to detect abnormality when you have no idea what that abnormality looks like, especially in the space of just how random and crazy the internet is all the time, identifying that is actually harder than it sounds. And what makes it potentially even harder is that a lot of the middle boxes that would be doing that detecting is exactly the middle boxes Geneva's mucking with with these strategies. So it may be the case that their detectors are also getting screwed up. Whatever, an imaginary detector would also be getting screwed up by these same strategies. So it's something they could take an action against. But we haven't seen any censors roll out something like this. Something else you could imagine, the existing fitness function we've just described for Geneva, it kind of assumes a static adversary, like an adversary that's not playing along, if you will. But it's also assuming an adversary that's not doing anything special to hunt it out. You could imagine a sensor that's a little more sophisticated than that. So something we've kept an eye on is, is at the end of the future, if either the sensor starts rolling out AI ML techniques, or if the sensor starts hunting for traffic that looks very abnormal. And you could imagine encoding additional bits into the fitness function, such that you could encourage Geneva to make this strategy blended with normal traffic. I want this to look as normal as possible, but still get through things like this. So you could imagine all sorts of modifications to the fitness function to make an algorithm like this a stronger competitor against an adversary that's also playing along. But we haven't seen the adversaries do that yet. So we haven't needed to. I was surprised when we talked to a bunch of, you know, also people in the intersection of security and machine learning that there are, as you say, these ML based, let's say, malware detectors or things like this, I guess also weird traffic detectors and people use them, for example, for company networks and so on. And these are, to my surprise, also, for example, vulnerable to adversarial attacks. So there's an entire new direction opening, which usually people imagine adversarial attacks like, I changed the image a little bit, and it's really this distinction between how the human sees it and how the machine sees it. But you know, in malware, it's like just bits and I flip like, you know, very small number of bits. There's nothing like how the human sees it and how the machine sees it. It's so weird. But yeah, I think I think it's pretty cool. And you got some attention in the media, and the articles usually go something like, this AI can evade censorship or something like this. And now knowing that you use genetic algorithms, what do you how do you think? How was how was your work received in the media? What do you think about it? Do you feel like they are kind of trying to put a few buzzwords in there? Or were you happy with it? In general, pretty happy. I've kind of been lucky to I mean, even just discussions like this, or we can talk about the work and then a deeper context than just like throwing buzzwords around. Like this is just an awesome way to kind of cut through that that buzzwordy fanfare, if you will. Yeah. So I've been kind of lucky. You're always going to see buzzwords attached to things that's always something like that. But I'd say overall, it's been it's been received positively and things like this are really what helped us get there. Cool. And the just saying the code for Geneva is available. It's on GitHub. Anyone can anyone can I guess look it up. Your builds fail right now. I just have to tell you I'm sorry. Yeah, we're switching between CI systems and haven't finished the migration. Okay. Yeah, nothing new here. So where is there I mean, there is a lot of open space here, it seems the genetic algorithms are very cool. They're like a basis right here. Do you think there are more places where like machine learning techniques, especially you said, you know, we kind of have to draw back from the gradient based approaches, but there are definitely there's definitely possibilities. If you think of something like, you know, AlphaGo or something like this, that's it's a discrete game. But also, you know, they they work with neural networks that, for example, when you build your tree, your modifications that guide that somehow that, you know, have an idea which of the modifications might lead to a better algorithm to a worse algorithm and so on. Do you see any sort of evolvement that could happen there? Definitely, definitely. When we first grow Geneva, our goal was not to be the last AI approach to the space. It was to be the first and hopefully the worst. It would be great if viewers out there, hey, take a crack at this. There's all sorts of new techniques out there just waiting to be applied. This space is rich and it's interesting and it's impactful. Like this is the kind of space where you discover something, get that out in the world, you're helping journalists and activists like right now. So we're really excited to see where this space goes and continues to blossom. So yeah, all sorts of all sorts of techniques just waiting to be applied. And are you also actively investigating the the censors side? Because I imagine that the more or the more capable you are in censoring things, also the better you can research counter strategies. So a bit. We've tried to tailor our research in such a way that we're not directly helping a sensor. We never want to publish a paper that's like really the use case of this is just making the sensors better. So if we do do research down that vein, it's purely in service of let's make invasion better. And we've tried to be very good about not releasing anything and not publishing anything that's directly, hey, censors, this new technique, man, that's going to really change the game for you. You should try and roll that out. So I guess that answers your question. Yeah. So what if you if you look ahead, you said, yeah, we said the space is wide open. What would be what do you see as a a, like maybe a bit of a north star for for the field, like for let's say censorship evasion or something like this, what would be characteristics of an ideal algorithm? That's a really good question. Ideal algorithm, something to shoot for, so I think I can answer that question by talking to I guess how this how the problem of censorship is getting harder and getting more complicated. So as censorship is continuing to evolve, like this this cat and mouse game exists, it's not just sensors patching bugs, like sensors themselves are flouty, getting more sophisticated, they're getting better. And one direction that we think sensors will start exploring in the future is this idea of more personalized censorship. So instead of censorship policies being rolled out for the entire country, you can imagine a system where users with elevated social credit scores or different professions, things like this could access different content online and be subjected to different different forms of censorship. And in cases like this, something like just directly applying Geneva gets a little bit harder because you can't just apply Geneva in one vantage point and help everybody, right? Like you need to suddenly have a way to to reach more people and help more people at once. So it's this question of how can we scale this up in a large way? And how can we scale this up safely in a way that protects itself from attacks from the adversary like the nations they can see our traffic. So in theory, they could muck with the training. How can we prevent that? So in crafting this like ideal algorithmic circumstances, a lot of things you have to consider. So I think building towards this idea of can we do federated training across a large a large population? Can we do this in a way that protects users? Can we make the algorithm more efficient so it needs it needs less connections to figure things out? All sorts of things like this, I think are really good goals to shoot for. And as more people viewers try this out, as more people like jump into the space and play with this, these are some of the problems they're going to be building towards. Is there any work on like screwing with the sensors? I imagine that if I you know, if I build an invasion attack that has like a really low hanging fruit of fixing it, and that fix in itself would somehow be, you know, completely devastating, but I don't know it when I implement it. Is there work in this direction? So is there work in the space of mucking with sensors? Definitely. Crafting the kind of attack you describe is kind of tricky because we don't know what the sensors code looks like. Yeah. Now there is this there is this idea of there are there are bugs and limitations that as they patch them may expose them to other attacks. So one quick example of this, if we go back to our analogy of we're sending letters back and forth, a common a common limitation that many less sophisticated sensors experience is they can't if I've taken a packet or taken a letter and I break into two letters, they can't put them back together. Yeah. Right. And that's that's like a huge limitation. It's really easy for me just to take a pack, split it up and send it through. So to fix that sensor, all it needs to do all it needs to do is remember every packet it sees and then stitch it back together based on the numbers on each of the packets. So that's like a simple fix to a limitation. But when you apply that fix, you open yourself up to the entire space of attacks of maybe I can sneak a letter in there that you think belongs halfway through the message, but it actually belongs to the beginning or actually belongs to the end or it actually doesn't belong in that at all. And so you have this is one example that we've seen in the wild where this idea of I have I need to fix the limitation and by fixing the limitation, I've opened myself up to a dozen other potential attacks. So that definitely exists. How how how I'm just thinking from my newbish understanding right here, how much of a problem is it that our protocols are rather fixed? I imagine if I could if I had like a dynamic language where if I communicate with anyone, the first step would actually be to negotiate a protocol in a very dynamic way, right, that would sort of give me the possibility much more to together with the person that I want to communicate with, negotiate something that could get around these sensors in a in a completely adaptive fashion. Is that at all feasible? Or is there some some flaw? So is it feasible? Maybe. I mean, if if such a thing like that could be built, it'd be incredible. It'd be awesome. So AI people, AI people watching get on that because that sounds that sounds awesome. There are definitely some challenges into into rolling that out. And you basically need to get in the headspace of if I roll out this protocol, and the sensor knows about it, what is it going to do? What is it going to do? But yeah, so there are there are protocols that exist out there where from the very first bite you sense the whole thing is encrypted. And in that case, it's pretty hard to fingerprint, right? It never looks the same. It's always just a stream of random looking bytes. But the sensor can also find that just by looking for something that looks like a random stream of bytes. And just like you said, that protocol never changes. It always looks the same. So if you you need to really develop a system that's flexible and dynamic enough that today it looks like this protocol, it's more it looks like this protocol today, it looks like nothing in between. So you really need to be very creative and very deliberate with how you do it. So I'm not aware of anything like that personally, maybe someone's working on it out there, but it would be awesome if you could do it. Now speaking of mocking with sensors, you also have other work that uses the censorship infrastructure. So essentially anything that's in place from the sensors to perform some some attacks, as I understand it, any any attack you could do is actually made potentially worse by the censorship infrastructure, such as a DDoS attack or something like this. Do you want to talk a little bit about that? I would love to. Yeah, so an area of work that we went that we started exploring a year or two ago, something we noticed a lot of these sensors is when you interact with them as a user, like they need to respond to you, they need to send you some traffic, right? Like if I'm if I'm trying to request some resource, and that resource is forbidden, maybe the sensor sends me a block page and that block page says, hey, you're not allowed to access this. And the thing is that that communication there, what's going on is my request can often be much smaller than the size of the block page I get back. So as an attacker, this opens up the space of hey, maybe I can use the sensor to launch an attack at somebody else by making a request for forbidden things, pretending to be someone else, and then letting them send that huge response at that other person. And this is this is an idea of a reflected attack or an amplification attack, because as an attacker, I can make a tiny request and get a bigger request out of it. So I'm amplifying my traffic. So amplification attack. So we started exploring whether we could do this to sensors and use these nation state sensors or even just beyond sensors, there's normal firewalls, like things that universities or just regular networked organizations have deployed. We discovered hundreds and hundreds, tens of thousands, millions of IP addresses that were behind these sensors that we could use to launch these attacks. And these attacks got crazy powerful. And the so the the who does it hurt more the sensors or the final recipients of the the attack? Yeah, so in this case, the weight is buried by both, but the brunt of the impact will be felt by the victim. Yeah, this line of work, it mucks with the sensor. But really, really, the some of the I want to say the purpose or something you can distill this work down to was sensors are causing more harm to the internet than they're not just the harm of a sensor is not just restricted to the citizens within its borders. Like a sensor anywhere is a threat to anyone everywhere. Yeah. So it's this work was less about let's flood a sensors network and more about let's prove to the world of these things are dangerous when they've been applied as carelessly as they've been deployed. Now other than block pages, you have some you have some very specific schemes of what you do specific to the censorship infrastructures that make these attacks even more powerful. What are examples of that? Yeah, so discovering these attacks in the first place, I'm making it sound very simple, right? You just send a request and then the response gets through. But I'm skipping over kind of an enormous step in here because what I've just described send a request pretending to be someone else should not be possible. Yeah, that that sentence should not exist. And it shouldn't be a thing you can do. And the reason that's the case is because when we make requests all the time, this happens I think there's a I think there's a gif in there that explains exactly what I'm saying. Just scroll up a little bit. There's a three way handshake that we need to complete. And that three way handshake is just this short exchange of packets. I think it's the one right above that. It's the short exchange of packets at the very beginning right here short exchange of packets that exists at the very beginning of our connection. And as an attacker, if I try and spoof a three way handshake, if I pretend to be my victim and start the handshake, the server is going to respond to the victim. And so I won't be able to get the critical bit of information I need from that handshake to finish it. And I need to finish that handshake in order to make a request. So throughout all of the all of networking history, basically up until this paper, it's been assumed that TCP, this underlying protocol behind all these requests is immune to these type of amplification attacks, largely immune. There's a small caveat there, but it's not worth getting into. So how do we go about addressing this problem? We used Geneva and AI techniques. And basically we replaced Geneva's fitness function and we told Geneva, hey, you can talk to these sensors, but instead of rewarding you for getting forbidden content, what we are going to do is we're going to reward you for getting content without establishing a connection and we're going to reward you for getting the biggest content you possibly can. So kind of turning the fuzz around its head a little bit and letting it explore the space of strategies that A, confuses the middle box into responding, so tricking it into thinking we have a connection already. And then B, once we've tricked it, getting the biggest possible response we can. And so this is a second set of work that was really powered by the same Geneva genetic algorithm. And we were able to use the same set of building blocks and primitives and programs that we had developed previously. We just applied them in a new way. And this is, if I understand it, it is not a weakness in TCP. Like if TCP were implemented correctly, Geneva wouldn't be able or shouldn't be able to find something around this, but this is specifically because these middle boxes are in there, right? Yeah, you're spot on. TCP itself is not the problem. It's the implementation of TCP. And that's partially why when we did this paper, we did this work, you can't just study TCP itself. You can't download the protocol specification, like think really hard, because that's not going to help you. We had to actually study real world sensors. So that's what we did. We took Geneva and we trained it against hundreds of sensors around the world. And then we took the results of that and were able to scan the whole internet. We scanned the internet almost 50 times actually, IPv4 internet, with these different packet sequences that Geneva discovered and effectively just attacked ourselves over and over and over again to see what kind of damage we could do. And how does that square? So before you said we're never going to release anything that helps the sensor in any way. And now you're releasing a recipe for launching massive attacks on something, right? I mean, I usually think any technology can be used for like with that, I could actually attack the sensor directly, right? And just make their life miserable using their own infrastructure, which is ironic even. I could use it to DDoS the Red Cross as well. So my perspective usually is that any technology can be used for good and for bad. But you've before said a little bit into the direction, we never want to publish anything that helps the sensor. This seems to be different. What's different here? Yes, the difference here is, and I want to note that we didn't just discover these and just immediately put them out into the world. We spent almost a year actually just doing responsible disclosure. We emailed every middle box manufacturer we could get in touch with and gave them advanced copies of our paper, advanced copies of this attack. We actually emailed, there's something called CERTs, Country Level Emergency Readiness Teams. These are teams that exist in various parts of the world that are basically designated to respond to network events pertaining to that region. So we emailed all of them around the world, so we were like, hey, that Chinese sensor you guys are operating, potential problem there. So we spent months and months working with DDoS manufacturers, CERTs, middle box manufacturers to try and patch these things and clean them up before this ever got out into the world. At the end of the day, this kind of runs into this broader responsible disclosure thing that a lot of the security field wrestles with of if I never publish this, there's often no incentive for this issue to be patched. Like if there's no downside to the network, they don't need to patch it. And if someone else discovers it before this gets out there, then they can start using it without the world and the defenders knowing about it. So there's this really tricky line you got to tow almost of I need to let everyone have as much time as possible to patch it, but they also need to know it's going to get out there to incentivize them to patch it. So with that in mind, we took the approach of let's take as long, as much time as we possibly can, let's tell everyone, any invested party about this attack, how to patch it, how to fix it. We gave them scripts to test their own network. And then after several months had passed and we were confident that they were, if they were going to take action, they already did, then we release the work. Cool. Yeah. Now you're a member of something that's called BreakerSpace. I've already mentioned it at the beginning. Do you want to maybe, because it's pretty unique, do you want to talk a little bit about what this is and what it does? Yeah, I'd be happy to. So BreakerSpace is a lab at the University of Maryland. Any UMD students watching, come check us out. The BreakerSpace lab, the kind of defining feature of this lab is that undergraduate students are invited to join and participate in the lab. So it's, the goal of this lab is to broaden and make research more accessible beyond just like PhD students and graduate students who are doing it. So this Geneva team and the broader censorship team within this lab has been staffed. I've been leading the team, but I've had a team of undergraduates who've been working with me on these projects. So every project we've talked about today and every paper on our website, this has not just been a one-man show. This has really taken a village to get these off the ground and get these moving. It's huge, huge tasks. And maybe you're missing, I didn't mention, a huge team of students who have been working on this with me. And okay, not unrelated to them being undergrads or not, did you, like how often does it happen that you get into like hot waters, like, you know, that there, you know, insecurity research, there are implicate, there are national defense implications, there are legal implications and so on. Like how do you navigate that space and how often does it happen that you're like, oops, I hope no one noticed this. It definitely, it definitely happens. And it's, we're really lucky to have such a supportive like university atmosphere in which we can do these things. We've worked closely with IRB, the Institution Review Board and our network security people. I mean, there was one week where we, for that scanning paper we were talking about, we're like, all right, let's kick off some scans. And then we immediately knocked out the university firewall. It's like, oh no. And they worked with us and helped us get it back and then helped work in such a way that wouldn't happen again. So what you're describing absolutely happens. I mean, one time we were accidentally, we didn't know this, we were accidentally attacking like the city of Jacksonville, Florida. And it was like, whoops, let's go email them. So that stops happening. Like the University of Kentucky, things like this. So what you're describing happens all the time. And it's like, oh shoot, whoops. And often those like whoops moments are like, that's a cool discovery you just made. We also got to go fix whatever you just broke. So totally happens, happens all the time. We've got lots of crazy stories like that. We're really lucky to have such a supportive atmosphere in which we can do these things. It's okay to break things as a work to fix them, obviously in such a supportive atmosphere. Where can people go if they want to get started in this space? Like let's say I'm an AI researcher. I want to have a good understanding of whatever reinforcement learning and evolutionary methods and genetic algorithms and all. But I've not much clue of security. Is there resources I can go to that you can recommend? So for security in general, there's so many, I mean, I'm sure there's two dozen YouTube channels that could probably hook you up with like incredible. So maybe we can send someone and link some of those below or something. I wish I could say that there is like this amazing AI censorship. I want to select censorship resource space where everyone can come to and learn how to apply AI to these techniques. Something like that doesn't quite exist, but there are great resources for learning about what censorship is happening in the world. So something like UNI. UNI is OONI. That's the Open Observatory of Network Interference. It's a spin out from the Tor team that monitors censorship all over the world. You can pull up the website later, but they can identify censorship in basically every country. It's run by volunteers and it's an incredible organization. So there's all sorts of groups like this that are studying censorship, monitoring for censorship. So for people who want to break into this more specific field of censorship, there's all sorts of great resources. Censored Planet is another group run by the University of Michigan. They're an awesome team. They also publish all their data. So all these groups have this very open sharing, hop on their website and they got lots of great resources, reports, data. You can get your hands in. Excellent. Is there anything else you want to get the word out to machine learning and AI people? Big open questions, anything that you feel should be out there? Especially just this whole space, this whole idea of there's this entire space of you can apply these techniques to in a way that's immediately impactful, helping real humans on the other side and humans who need this help. You have this potential to make a real immediate impact on the world. So it's a great space to get involved in. Excellent. Kevin, thank you so much for being here and bringing this a bit closer. I know more, I hope everyone else does too now. Thanks so much for having me. This has been a blast. Excellent. Super appreciate it. 스포ated Adams How awesome was that?
 Hello there, today I'm talking to Kevin Bok, who is a cybersecurity expert and one of the
 main people involved in the Geneva project.
 Geneva is a genetic algorithm that evades censorship by nation states.
 So in real time, Geneva can evolve to the ever more present danger of censorship by
 really big entities such as governments.
 All of this is done through an evolutionary search over a program grammar.
 And in this interview, we're going to touch on a whole range of topics including Geneva,
 how it works, what it does, why people research it and what it has done so far in the world,
 but also the broader topics of security and its connections to AI, how people can get
 started in this field and what the main questions and problems are in this space.
 Further, Geneva comes out of a project at the University of Maryland called Breaker Space,
 which is a sort of lab that includes undergraduates in security research, which is a really cool
 project.
 And I think highlighting this would be helpful to some people.
 Maybe you're at the university, you don't know this exists.
 Go there, take part.
 All right, without further ado, I want to give over to the interview and have fun.
 All right, everyone, I have with me today here Kevin Bok, who is a PhD student at the
 University of Maryland, a cybersecurity researcher, and a member of Breaker Space, which is a
 pretty cool project at the University of Maryland.
 He also has been in the news a little bit with a project that's called Geneva, which
 uses genetic algorithms to evade censorship by nation states.
 And I think that's pretty cool.
 So Kevin, welcome to the show and thanks for being here.
 Thank you for having me.
 I'm excited to be here.
 So the goal of today, it's a little bit different because I'm a total noob at security.
 Most of the audience of this channel is into machine learning.
 Maybe some know about security, some know about the censorship apparatus that's in place
 around the world and what people do about it.
 I think most won't.
 So today I'll be asking mostly noobish questions and we'll have you here to guide us through
 everything, to guide us through what's happening in this world.
 So maybe you first can start off a little bit.
 How did you get into, how did you get to the place where you are?
 What's the main things in security right now that draw you to it?
 I think security and the censorship space also is in this really cool time where AI
 and ML techniques have been exploding in all these other fields and they're just over the
 last four years really breaking into security and we're still figuring out all the different
 applications where you can apply these techniques in security.
 There's new techniques and new applications that people are discovering all the time from
 better ways to detect spam and better ways to identify, hey, this domain is malicious
 or AI-based scanners for that binary you downloaded, that's probably malware, things like that.
 So security field is still discovering all sorts of new ways you can apply these techniques
 and that was one of my motivations initially actually of bringing this to censorship because
 this project was really the entire field of censorship's first foray into using AI and
 ML-like techniques.
 And if you talk about censorship, what do you mean exactly by that?
 Yes, there's so many forms of censorship in effect around the world today.
 I mean everything from political pressure to self-censorship to taking down...
 Like there's so many different types.
 So I'm going to scope this discussion down a little bit, just the type of censorship
 that we study in this lab and that's this type of automated censorship that happens
 in the network performed by nation states.
 So what do I mean by this?
 If you're a user in certain regimes around the world, let's say in Iran or something
 and you try and make a request, as that request, as that web traffic crosses through the border
 of the country, it is scanned, parsed and inspected by some machines that physically
 reside in the network called middle boxes, because they're in the middle of the network.
 And these middle boxes examine your request and they say, is this something we should
 allow or not?
 And if the answer is no, they either inject traffic to take down your connection or they
 drop your connection or they do something to disrupt what's going on.
 And you'll notice everything I just said there, there's no human in the loop.
 There's no human content review or anything like this.
 It's a purely automated run by these middle boxes or firewalls deployed by these nations
 that just automatically inspect the internet traffic as they go by.
 So that's really the scope of what we've been studying here.
 Naive question.
 Why can't I just encrypt my traffic and then every traffic looks the same towards the outside?
 Yeah, that's a great question.
 So why can't we just encrypt everything?
 People have been trying.
 So there's like a couple of different approaches to this.
 You're like, well, let's just use HTTPS, right?
 Encrypted.
 We're good.
 Unfortunately, HTTPS has a small privacy leakage.
 When you first set up an HTTPS connection and that very first initial is called a handshake
 and that first back and forth, you as the client, as a part of the protocol, you have
 to announce the domain you're talking to.
 And that announcement happens unencrypted.
 So if you're making a HTTPS handshake to Wikipedia, in the very first packet you send, it's going
 to include the word Wikipedia.
 And that's called the server name indication field.
 You indicate to the server what the name of the server you're trying to talk to.
 And unfortunately, sensors just read that fields and then they take down your connection
 if you talk to a forbidden domain.
 So HTTPS, unfortunately not close, but not quite finishing the job.
 Now, I will say there have been just a quick sidebar.
 There have been some advancements in HTTPS to try and fix this.
 There's a recent proposal to encrypt that fields.
 It's called encrypted SNI.
 And China just started censoring that last year.
 So you can try and encrypt things, but these sensors are often just hostile to the idea
 of just letting their citizens just encrypt all their traffic.
 I guess it's a little bit like if everyone encrypts, like with HTTPS nowadays, everyone
 does it.
 So you can't conceivably block HTTPS just because you don't like some traffic.
 But if there's a new type of encryption, it's probably only the people that have something
 to hide that use that type of encryption.
 So is a strategy that the rest of the world as fast as possible would use these techniques
 to kind of make that approach unusable?
 That's exactly right.
 The broader topic you're actually discovering and saying out loud here is this idea of collateral
 damage, of can we make a protocol or something so popular and use so diversely that if a
 sensor were to try and block it, it would cause irreparable harm to good services.
 There's some meaningful cost to performing that censorship.
 So just like you've identified HTTPS, that's everywhere.
 They can't just shut down all HTTPS.
 But rolling out a new encryption method for HTTPS that's not very widely deployed, they
 can nip that in the bud and prevent its rollout.
 So there's kind of this interesting race in a game between developers and these sensors
 that's still being played out.
 Now let's talk about more, let's say, naive approaches.
 What is the development of the field?
 What has been tried before and what has been, let's say, thwarted?
 Or what's the cat and mouse game looked like in the past?
 I imagine different things like there's Tor, there is all kinds of things.
 There is probably things that everyone installs on their end, like VPNs and tunnels and so
 on.
 What's been the general development over the years?
 Yeah, so the researchers and sensors have been playing this cat and mouse game for two
 decades now.
 And it's kind of evolved and it's been playing out in multiple fronts.
 So you're exactly right.
 Tor has been a huge front on that war, if you will.
 We've developed Tor and continue to advance it.
 Unfortunately, there are some limitations, just the Tor protocol and sensors can enumerate
 the Tor entry points basically and just block you.
 So once you get into Tor, you're generally great, but they try and block you out.
 There's been all sorts of techniques people have proposed, like maybe I can disguise my
 traffic to look like Skype.
 And then the sensor's like, well, you didn't disguise it quite well enough, blocked.
 There's a whole interesting field of defeating censorship or subfield, I should say, called
 packet manipulation based censorship.
 And this is this idea where all our communication is happening via packets.
 And if you just tweak those packets in just the right way, you could cause the sensor
 to miss you.
 And historically, that's also been something that's played out in this cat and mouse game
 where researchers will study these sensor systems and then they'll find a loophole and
 they'll deploy it and use it.
 And then the sensor's like, oh, I'll fix that.
 And then we're back to square zero.
 So this game has really been continuing to play.
 I'll call one thing out real quickly about VPNs.
 Because a lot of people, particularly those who have been to China, are like, I've been
 able to use a VPN and it's been OK.
 VPNs in many places work.
 In many places they don't.
 There's a country in the news recently.
 They were in the news because they rolled out a new law that forced their citizens to
 swear on the Quran that they would not use a VPN in order to get internet access installed
 in their homes.
 It's just like crazy sentence to say out loud.
 But in China, for example, these VPNs, many of them work most of the time.
 But what researchers have noticed is that around the time politically sensitive events
 are happening or political, such as elections, things like this, a lot of VPNs will just
 mysteriously stop working.
 And then after the event, they'll mysteriously start working again.
 And it kind of points to this broader idea that some of these countries may be sitting
 on more censorship capability than they deploy on a daily basis.
 And they have more power than they use.
 So this cat and mouse game may even be stronger than we think it is.
 Can you give us an idea of what this packet manipulation evasions look like?
 Because I imagine something you mentioned before, if there's Wikipedia in the header,
 I don't want my population to see Wikipedia.
 Like that's it.
 What can I possibly manipulate there in order to get through such censorship?
 Yeah.
 So we can think about sensors as our computers are sending packets around.
 You can imagine a lot of that communication like you're writing mail, your packets are
 envelopes that are going to the network.
 And in order to have a communication with a server like Wikipedia, that's going to take
 a couple of envelopes back and forth.
 And the sensor is just like the postman in the middle reading all your letters.
 And unfortunately that postman has got to process a lot of letters, a lot of letters.
 And you can imagine something the scale of like China, you're dealing with a huge, huge
 volume of traffic just at a constant basis.
 What that means is the sensor can't just remember everything it sees.
 So for example, if it's trying to track that, hey, that person over there is trying to talk
 to that server over there and that person over there is talking to that server over
 there, that state it has to maintain.
 And the amount of state it has to maintain, it'll grow.
 And the size of some work like China, it could grow pretty fast.
 So they have to be really careful about what they remember and the state they maintain.
 So you could imagine doing something like, let's say we're exchanging packets.
 There exists a type of packet called the reset packet.
 And these are normal packets our computers send these all the time.
 But they basically just exist to tell the other side, stop talking to me immediately.
 I'm hanging up the connection.
 So you can imagine doing something like you and I are communicating, we're sending these
 packets back and forth.
 And I just slip one additional packet into the connection towards the beginning and it's
 a reset packet.
 And I'll send that packet along.
 And when the postman sees that packet, he's like, well, these guys have stopped communicating
 after this message, he's going to ignore him forever.
 And then he throws away the state he's maintaining about our connection.
 He forgets that we're talking because why would he need to remember anymore?
 He thinks we're done.
 And if I craft that packet in such a way that it won't make it to you, or you'll see it
 and ignore it or something like this, then we'll be able to still communicate fine, right?
 Or our communication is unimpacted.
 But any of the packets that go by, the sensor's like, I don't know who this is.
 And you can get through.
 So this is like the broad strokes, this idea of packet manipulation based censorship, where
 you're tweaking the packets that go by to try and basically trick the sensor that's
 in the middle into letting you continue to talk.
 Now do I see this correctly, that there have been like a giant amount of these schemes
 proposed and as you say, there's a cat and mouse game.
 One is being proposed, then they fix it, then another one, then they fix it.
 So that points to the possibility of what if we could have something dynamic, right?
 What if we could have something that by itself tries to invent new things?
 And that's where you went with Geneva.
 Do I understand that correctly?
 That's exactly correct.
 Yeah, you're spot on.
 Yeah, so over the years, there's been, I want to say dozens of these that have been proposed
 and researchers have, it's exactly this cat and mouse game.
 They studied the censorship system.
 I mean, the censorship system is not public, so they're probing it, they're trying to take
 measurements.
 That's a lot of work.
 And then they get an understanding, they apply their good human intuition, they develop something
 cool and publish it and the sensor fixes it.
 They don't tell you they fixed it.
 They don't publish a paper that's like, hey, we just fixed your bug.
 So it just resets this to square zero.
 And so the idea with Geneva, which stands for genetic invasion, the idea of this was
 it's an algorithm that could kind of flip this process on its head.
 So instead of a human having to take the approach of let's understand how the censorship works
 and then defeat it, let's just have some AI or fuzzer or automated system, just attack
 the sensor, figure out ways through and then give it to the human.
 And now after the fact, my slow human brain can go figure out why that thing works.
 And now my brain is no longer the bottleneck to helping people get through the sensor.
 How does this, you want to go a bit more into detail?
 I mean, it sounds great at the surface, but there's a reason, right?
 We need security researchers probing, making sense.
 And there's a reason that's the bottleneck.
 If I were just to be like, well, you know, fuzz a bit, it's probably not going to work.
 So what does Geneva do that allows it to even be successful where maybe humans take a long
 time or wouldn't be successful?
 Yes, there were a couple of pretty significant challenges when we first started in applying
 something like a genetic algorithm or really any AI to the space of censorship.
 And if you think about the way censorship works, it's not hard to imagine like why that's
 the case.
 Because if you think about think about a censorship problem, right, like a query is either censored
 or it's not, it's just a binary decision.
 So it's not like your traditional ML or AI where you have this nice like gradient descent.
 There's no error.
 You're back from the sensor.
 The sensor doesn't tell you like, hey, if you tweak your query, just a little bit, you're
 getting closer.
 Yeah, you know, there's no gradient which with which you could work.
 So that that property alone rules out the majority of the ML field as far as approaches
 you can take.
 Is there even a loss?
 Like you said, it's hard to detect if you even get through.
 How do you do that in the first place?
 How do you notice success or failure?
 Yeah, so in our case, you're exactly right.
 Capture capturing that can be difficult.
 What we do to make it easier in ourselves is we obtain machines inside these censored
 countries and directly try to request for written content.
 So Geneva trains directly against the sensor and we know we got it.
 When the sensor takes action is kind of obvious.
 So Geneva will try and obtain some forbidden content while manipulating the packet stream.
 And then if it succeeds, great.
 If it fails, we'll know.
 Right.
 So this idea of how do we apply ML, AI, some fuzzing to this space?
 Like how do we build to this?
 There's a couple of main challenges towards doing that.
 The first is this total lack of gradient that I mentioned.
 And really that only leaves you with kind of a small number of approaches.
 And we chose to go down the route of let's use a genetic algorithm for this.
 There's some nice properties.
 It's easily explainable.
 You can understand how it works while it runs.
 It's a little less black boxy than something more like a neural net or something or Markov
 or something like this.
 But if you want to build a genetic algorithm, you need a couple of things.
 You're seeing what some of these strategies look like right here.
 So if you want to build a genetic algorithm, there's a couple of things you need.
 You need some building blocks.
 Something that the algorithm can compose and put together.
 And you need some way for it to put those things together.
 I mean, us humans as examples, as far as genetics goes, we've got our DNA bases, right, ACTG.
 And we can put those together in DNA.
 For the genetic algorithm for Geneva, we needed to decide what makes sense for building blocks
 for the algorithm to use.
 And that alone is like an initial really huge challenge because you could be creative and
 you can think about a million different ways an algorithm could manipulate a packet, right?
 Flip a bit.
 You could flip this bit.
 Like there's just so many different things you could give it to do.
 So one of the first challenges we had to figure out was how do we balance what this algorithm
 can and cannot do to the data it has?
 And on one hand, we could let it flip any bit.
 The downside of that is it could take forever to learn to check some, but it's super powerful.
 Like on the other extreme there, we could just encode what previous researchers found
 and let it play with those together.
 It would be super fast, but it'd be hard to learn anything new, right?
 We'd just be building in biases directly.
 So the approach we ended up taking was giving Geneva basically the same ability to change
 traffic as what the network itself could do.
 So the network itself has just a few set primitives that can do the packets.
 It can take a packet, make multiple packets, it can duplicate them, it can change a header
 to something, it's tampering a packet.
 You can take a packet, break it into multiple pieces, fragmenting.
 You can take a packet, drop it, which is just basically deleting the packet.
 So we built out these building blocks and then allow it to compose these things together
 in trees.
 So like syntax, you give it a syntax and it can assemble a little program out of this
 syntax, like one we see right here.
 That's exactly correct.
 Can you walk us through what this particular thing does?
 Sure, sure.
 This is kind of a fun strategy.
 So there's a few different components to a Geneva strategy.
 I'll break down the syntax for you real fast, what these programs look like.
 So the first component is the idea of a trigger.
 The trigger is what's between the square brackets.
 So there's two triggers in this, TCP flags S and TCP flags R. And when Geneva is monitoring
 traffic, the trigger tells it which packet should I act upon.
 So this first trigger you see here says TCP flags S. So that means that whatever actions
 are attached to that trigger will run on any SYN packet it sees.
 S stands for SYN.
 SYN means the start of my connection.
 So what this is going to do to that packet is the very first action we see is duplicate.
 So that means it's going to take that packet and make two of them.
 Now duplicate, the syntax of this is it's one set of actions, comma, another set of
 actions.
 So you'll see the two actions you see here are tamper and then send.
 So the second duplicate we do nothing to.
 So the second duplicate we're just going to send on the wire.
 But to the first duplicate what we're going to do is we're going to replace the flags
 fields in that packet with SYNAC SA.
 And then we're going to send that packet.
 So basically what this little program does is it sees outgoing SYNAC packets, outgoing
 SYN packets to your computer, and it duplicates them to make two packets and then replaces
 the flags in the first one with SYNAC.
 Now any networking person listening is like, this is clearly ridiculous.
 This never should work.
 Why would we even do this?
 Why are we talking about this?
 And what's going on here is that for certain sensors around the world, SYNAC is the packet
 that's typically sent by a server.
 It's never sent by a client.
 So what's going on in this strategy is when the client sends a SYNAC, the sensor says,
 whoa, I must have missed something.
 This client is clearly a server, which means the server must be the client.
 It reverses the roles of client and server in the mind of the sensor.
 And as a consequence, when the client makes the real request, since the sensor is processing
 packets differently between client and server, you're through.
 I see.
 So that's this idea of the strategy.
 So that connection in the mind of the sensor is already established as here's a server,
 here's a client, and it kind of keeps that state for subsequent packages.
 More or less.
 Yeah, that's exactly it.
 So this is an example of just one strategy in one of these programs that...
 So Geneva built this program itself and it built this through the process of evolution.
 And you've discovered, just to jump ahead a little bit because we're not through yet
 with explaining exactly how it works.
 But you've discovered that Geneva will actually reproduce a lot of the common or known or
 already discovered things that researchers have proposed, right?
 Yeah, we had this really cool result initially where we set out to try and...
 We wanted to, when we first developed this tool, kind of benchmark it against the rest
 of the fields.
 And that's kind of challenging because sensors have continued to evolve.
 So what we did was we sat down in the lab and we implemented in the lab our best guess
 as to what...
 Our best implementation, I should say, as to what these sensors looked like based on
 what previous researchers found.
 And then trained Geneva against these mock sensors and also trained it against the great
 firewall and real sensors where we could.
 And we found it was very quickly, it was able to reproduce basically the entire field.
 Every strategy a human had come up with, this also found and it found them pretty quickly.
 So it's really showing the power of automated approaches and AI ML.
 So you have...
 Let's get back a little bit.
 You have this syntax, right?
 That you can build trees from which are valid programs in Geneva.
 This will modify the traffic somehow.
 Now to say that most of this traffic will just not even be traffic probably, like the
 connection will be somehow bad.
 Some of it will go through and some of it will actually maybe evade the sensor.
 What do we need to get there?
 What do we need to get to a place where...
 I guess if you just do it naively and you randomize a little bit, it will just be bad.
 Like 99.9% of all the programs you generate, you'll initiate them and then after a while
 you'll see like my traffic isn't even getting anywhere, right?
 So what are the...
 Of the genetic algorithm components, what do we still need?
 Yeah.
 So we're building our way up to the genetic algorithm.
 We've got, just like you said, we got our building blocks.
 We got a way to put them together.
 We got a syntax so we can build these programs out of it.
 We can run these programs on network traffic.
 And you're exactly correct that if we initialize completely randomly, it's going to do terribly.
 And that's exactly what happens.
 We've tested this.
 So where do we need to go from here now that we have this?
 So this kind of brings us to this idea of let's get evolution in the mix.
 So you can imagine the way this works is we have a big pool of strategies.
 Okay, we'll call this a population.
 And each of these populations just take for granted for now that we have some diverse
 set of strategies in here.
 And we have a way to test them, right?
 We can try and make requests for something forbidden and we can run these programs on
 those requests as we make them.
 So for example, from inside of China, we can try and access Wikipedia.
 That's a sensitive resource.
 And we'll have these programs running on that connection.
 We'll just try and make that connection over and over again.
 And what we'll see is some of these strategies will destroy our connection.
 Some of them will just not work at all and do terribly.
 Some of them might keep our connection alive.
 And maybe if we get crazy lucky, we'll defeat censorship.
 But for now, let's just say a whole bunch of them will just destroy our connection and
 maybe some won't.
 We have is a fitness function.
 And this fitness function, this is a bar, a much broader space in ML and AI, but it's
 basically this idea of if you take some individual from the population, some individual strategy,
 how good is this thing?
 Survival of the fittest, like should this thing survive basically and continue to propagate
 its genetic material?
 So this was actually the second big challenge in applying AI and ML to this space of censorship
 vision of what on earth should a fitness function look like in this space?
 Because just like we talked about earlier, there's no gradient, right?
 And even come up with like a loss function can be a little tricky.
 And I mean, even if like, sorry to interrupt, but the fitness even like if the fit, I guess
 the fitness, is it anything else than zero?
 Like, okay, maybe some connections don't even work to like the server next to you.
 You can discard those.
 But other than that, the fitness is either doesn't reach the target or does reach the
 target.
 And if it does, you've kind of won, right?
 Like how can you even get a meaningful signal?
 Is there a fitness in between zero and one?
 Yeah, so and part of what makes Geneva work is we've kind of shoehorned our way to getting
 fitness between zero and one.
 And specifically what we do is rule out those strategies that break your own connection.
 So that's kind of how we've gotten between zero and one.
 Because it's not technically zero and one.
 It's almost negative one, zero, one.
 And negative one is Geneva shooting itself in the foot, right?
 It's just like dropping all your traffic.
 That's never going to work.
 And we shouldn't even bother exploring that space more, right?
 Like we're never going to go anywhere.
 But if you can make it so that your packets are at least interacting with the sensor and
 at least have the potential link to the server, well, now we might be getting somewhere.
 So basically what we do is we set up the fitness function in such a way that if strategies
 destroy the underlying connection, they'll be punished severely and basically killed
 off.
 And strategies that interact with the sensor, even though they get censored, they'll get
 a slightly higher fitness function than those other ones.
 So what's going to happen is because those individuals aren't, they're not successful,
 but they're still the most successful in the population pool, which means some subset of
 them will continue to reproduce.
 And basically that subset is just chosen randomly.
 But because we're just choosing randomly, mutation is still going to happen.
 So we're basically taking a set of individuals, they all interact with the sensor, and then
 we just mutate them and try again, and then mutate them and try again.
 And effectively what this has turned into is a fuzzer.
 Like Geneva is, the fitness function basically makes this a targeted fuzzer where we can
 fuzz just the space of strategies, just the space of programs that allow us to interact
 with the sensor.
 And then where it gets interesting is as this fuzzer is running generation after generation,
 just trying different crazy things against the sensor, if it finds something that gets
 through, suddenly that fitness is way higher than everything else.
 And that individual will start sharing its genetic material and propagating within the
 population pool.
 At that point, we could stop.
 We could stop the fitness function right there.
 But we optionally add some additional punishments and rewards for the algorithm at this point.
 And specifically we add basically a punishment for strategy complexity.
 So if an individual is successful, we optionally punish it for basically the number of actions
 and the amount of overhead it adds to the connection.
 And the reason we do that is this is not strictly required, but I have a very small, smooth
 human brain and it's so much easier to understand a strategy that's only two actions long,
 compared to some that's 50 actions long, for example.
 So if we could encourage the algorithm to be like, great, you got a solution, now simplify
 it down for me.
 And it will over the course of generations whittle it down to its smallest form and then
 at the end present to you its population pool and its best individuals.
 And we see here a few ways you can mutate.
 I think this just essentially comes down to changing the syntax tree in some form.
 Yep.
 And you can imagine all the different ways you can take these programs and mix them around.
 If you can think about it, Geneva can probably do it.
 And so just maybe for my understanding, but you're trying all of this, you say you have
 some machines inside of these countries.
 And I read some like, obviously this is not going to work against IP blocking.
 How do you not get IP blocked by them?
 I imagine there's some weird traffic that hits my censorship wall all the time.
 Why don't I just be like, well, gone.
 Yeah, that's a good question.
 And we get this question a lot, actually.
 And you're pointing to this broader question of what's the censor's response?
 You're doing all these wacky, crazy, ridiculous things.
 There's a strategy in there that just lights up every TCP flag.
 That package shouldn't exist flatly.
 It has no meaning on the network.
 But Geneva tried it, found it, and found that it works.
 So where do censors go from here?
 It sounds like, when we're talking about things like it's sending crazy packets, it sounds
 like that should be something that's easy to detect on the network.
 But it sounds easy until you try and write it.
 Because if you think about it, writing something to detect abnormality when you have no idea
 what that abnormality looks like, especially in the space of just how random and crazy
 the internet is all the time, identifying that is actually harder than it sounds.
 And what makes it potentially even harder is that a lot of the middle boxes that would
 be doing that detecting is exactly the middle boxes Geneva's mucking with with these strategies.
 So it may be the case that their detectors are also getting screwed up.
 Whatever, an imaginary detector would also be getting screwed up by these same strategies.
 So it's something they could take an action against.
 But we haven't seen any censors roll out something like this.
 Something else you could imagine, the existing fitness function we've just described for
 Geneva, it kind of assumes a static adversary, like an adversary that's not playing along,
 if you will.
 But it's also assuming an adversary that's not doing anything special to hunt it out.
 You could imagine a sensor that's a little more sophisticated than that.
 So something we've kept an eye on is, is at the end of the future, if either the sensor
 starts rolling out AI ML techniques, or if the sensor starts hunting for traffic that
 looks very abnormal.
 And you could imagine encoding additional bits into the fitness function, such that
 you could encourage Geneva to make this strategy blended with normal traffic.
 I want this to look as normal as possible, but still get through things like this.
 So you could imagine all sorts of modifications to the fitness function to make an algorithm
 like this a stronger competitor against an adversary that's also playing along.
 But we haven't seen the adversaries do that yet.
 So we haven't needed to.
 I was surprised when we talked to a bunch of, you know, also people in the intersection
 of security and machine learning that there are, as you say, these ML based, let's say,
 malware detectors or things like this, I guess also weird traffic detectors and people use
 them, for example, for company networks and so on.
 And these are, to my surprise, also, for example, vulnerable to adversarial attacks.
 So there's an entire new direction opening, which usually people imagine adversarial attacks
 like, I changed the image a little bit, and it's really this distinction between how the
 human sees it and how the machine sees it.
 But you know, in malware, it's like just bits and I flip like, you know, very small number
 of bits.
 There's nothing like how the human sees it and how the machine sees it.
 It's so weird.
 But yeah, I think I think it's pretty cool.
 And you got some attention in the media, and the articles usually go something like, this
 AI can evade censorship or something like this.
 And now knowing that you use genetic algorithms, what do you how do you think?
 How was how was your work received in the media?
 What do you think about it?
 Do you feel like they are kind of trying to put a few buzzwords in there?
 Or were you happy with it?
 In general, pretty happy.
 I've kind of been lucky to I mean, even just discussions like this, or we can talk about
 the work and then a deeper context than just like throwing buzzwords around.
 Like this is just an awesome way to kind of cut through that that buzzwordy fanfare, if
 you will.
 Yeah.
 So I've been kind of lucky.
 You're always going to see buzzwords attached to things that's always something like that.
 But I'd say overall, it's been it's been received positively and things like this are really
 what helped us get there.
 Cool.
 And the just saying the code for Geneva is available.
 It's on GitHub.
 Anyone can anyone can I guess look it up.
 Your builds fail right now.
 I just have to tell you I'm sorry.
 Yeah, we're switching between CI systems and haven't finished the migration.
 Okay.
 Yeah, nothing new here.
 So where is there I mean, there is a lot of open space here, it seems the genetic algorithms
 are very cool.
 They're like a basis right here.
 Do you think there are more places where like machine learning techniques, especially you
 said, you know, we kind of have to draw back from the gradient based approaches, but there
 are definitely there's definitely possibilities.
 If you think of something like, you know, AlphaGo or something like this, that's it's
 a discrete game.
 But also, you know, they they work with neural networks that, for example, when you build
 your tree, your modifications that guide that somehow that, you know, have an idea which
 of the modifications might lead to a better algorithm to a worse algorithm and so on.
 Do you see any sort of evolvement that could happen there?
 Definitely, definitely.
 When we first grow Geneva, our goal was not to be the last AI approach to the space.
 It was to be the first and hopefully the worst.
 It would be great if viewers out there, hey, take a crack at this.
 There's all sorts of new techniques out there just waiting to be applied.
 This space is rich and it's interesting and it's impactful.
 Like this is the kind of space where you discover something, get that out in the world, you're
 helping journalists and activists like right now.
 So we're really excited to see where this space goes and continues to blossom.
 So yeah, all sorts of all sorts of techniques just waiting to be applied.
 And are you also actively investigating the the censors side?
 Because I imagine that the more or the more capable you are in censoring things, also
 the better you can research counter strategies.
 So a bit.
 We've tried to tailor our research in such a way that we're not directly helping a sensor.
 We never want to publish a paper that's like really the use case of this is just making
 the sensors better.
 So if we do do research down that vein, it's purely in service of let's make invasion better.
 And we've tried to be very good about not releasing anything and not publishing anything
 that's directly, hey, censors, this new technique, man, that's going to really change the game
 for you.
 You should try and roll that out.
 So I guess that answers your question.
 Yeah.
 So what if you if you look ahead, you said, yeah, we said the space is wide open.
 What would be what do you see as a a, like maybe a bit of a north star for for the field,
 like for let's say censorship evasion or something like this, what would be characteristics of
 an ideal algorithm?
 That's a really good question.
 Ideal algorithm, something to shoot for, so I think I can answer that question by talking
 to I guess how this how the problem of censorship is getting harder and getting more complicated.
 So as censorship is continuing to evolve, like this this cat and mouse game exists,
 it's not just sensors patching bugs, like sensors themselves are flouty, getting more
 sophisticated, they're getting better.
 And one direction that we think sensors will start exploring in the future is this idea
 of more personalized censorship.
 So instead of censorship policies being rolled out for the entire country, you can imagine
 a system where users with elevated social credit scores or different professions, things
 like this could access different content online and be subjected to different different forms
 of censorship.
 And in cases like this, something like just directly applying Geneva gets a little bit
 harder because you can't just apply Geneva in one vantage point and help everybody, right?
 Like you need to suddenly have a way to to reach more people and help more people at
 once.
 So it's this question of how can we scale this up in a large way?
 And how can we scale this up safely in a way that protects itself from attacks from the
 adversary like the nations they can see our traffic.
 So in theory, they could muck with the training.
 How can we prevent that?
 So in crafting this like ideal algorithmic circumstances, a lot of things you have to
 consider.
 So I think building towards this idea of can we do federated training across a large a
 large population?
 Can we do this in a way that protects users?
 Can we make the algorithm more efficient so it needs it needs less connections to figure
 things out?
 All sorts of things like this, I think are really good goals to shoot for.
 And as more people viewers try this out, as more people like jump into the space and play
 with this, these are some of the problems they're going to be building towards.
 Is there any work on like screwing with the sensors?
 I imagine that if I you know, if I build an invasion attack that has like a really low
 hanging fruit of fixing it, and that fix in itself would somehow be, you know, completely
 devastating, but I don't know it when I implement it.
 Is there work in this direction?
 So is there work in the space of mucking with sensors?
 Definitely.
 Crafting the kind of attack you describe is kind of tricky because we don't know what
 the sensors code looks like.
 Yeah.
 Now there is this there is this idea of there are there are bugs and limitations that as
 they patch them may expose them to other attacks.
 So one quick example of this, if we go back to our analogy of we're sending letters back
 and forth, a common a common limitation that many less sophisticated sensors experience
 is they can't if I've taken a packet or taken a letter and I break into two letters, they
 can't put them back together.
 Yeah.
 Right.
 And that's that's like a huge limitation.
 It's really easy for me just to take a pack, split it up and send it through.
 So to fix that sensor, all it needs to do all it needs to do is remember every packet
 it sees and then stitch it back together based on the numbers on each of the packets.
 So that's like a simple fix to a limitation.
 But when you apply that fix, you open yourself up to the entire space of attacks of maybe
 I can sneak a letter in there that you think belongs halfway through the message, but it
 actually belongs to the beginning or actually belongs to the end or it actually doesn't
 belong in that at all.
 And so you have this is one example that we've seen in the wild where this idea of I have
 I need to fix the limitation and by fixing the limitation, I've opened myself up to a
 dozen other potential attacks.
 So that definitely exists.
 How how how I'm just thinking from my newbish understanding right here, how much of a problem
 is it that our protocols are rather fixed?
 I imagine if I could if I had like a dynamic language where if I communicate with anyone,
 the first step would actually be to negotiate a protocol in a very dynamic way, right, that
 would sort of give me the possibility much more to together with the person that I want
 to communicate with, negotiate something that could get around these sensors in a in a completely
 adaptive fashion.
 Is that at all feasible?
 Or is there some some flaw?
 So is it feasible?
 Maybe.
 I mean, if if such a thing like that could be built, it'd be incredible.
 It'd be awesome.
 So AI people, AI people watching get on that because that sounds that sounds awesome.
 There are definitely some challenges into into rolling that out.
 And you basically need to get in the headspace of if I roll out this protocol, and the sensor
 knows about it, what is it going to do?
 What is it going to do?
 But yeah, so there are there are protocols that exist out there where from the very first
 bite you sense the whole thing is encrypted.
 And in that case, it's pretty hard to fingerprint, right?
 It never looks the same.
 It's always just a stream of random looking bytes.
 But the sensor can also find that just by looking for something that looks like a random
 stream of bytes.
 And just like you said, that protocol never changes.
 It always looks the same.
 So if you you need to really develop a system that's flexible and dynamic enough that today
 it looks like this protocol, it's more it looks like this protocol today, it looks like
 nothing in between.
 So you really need to be very creative and very deliberate with how you do it.
 So I'm not aware of anything like that personally, maybe someone's working on it out there, but
 it would be awesome if you could do it.
 Now speaking of mocking with sensors, you also have other work that uses the censorship
 infrastructure.
 So essentially anything that's in place from the sensors to perform some some attacks,
 as I understand it, any any attack you could do is actually made potentially worse by the
 censorship infrastructure, such as a DDoS attack or something like this.
 Do you want to talk a little bit about that?
 I would love to.
 Yeah, so an area of work that we went that we started exploring a year or two ago, something
 we noticed a lot of these sensors is when you interact with them as a user, like they
 need to respond to you, they need to send you some traffic, right?
 Like if I'm if I'm trying to request some resource, and that resource is forbidden,
 maybe the sensor sends me a block page and that block page says, hey, you're not allowed
 to access this.
 And the thing is that that communication there, what's going on is my request can often be
 much smaller than the size of the block page I get back.
 So as an attacker, this opens up the space of hey, maybe I can use the sensor to launch
 an attack at somebody else by making a request for forbidden things, pretending to be someone
 else, and then letting them send that huge response at that other person.
 And this is this is an idea of a reflected attack or an amplification attack, because
 as an attacker, I can make a tiny request and get a bigger request out of it.
 So I'm amplifying my traffic.
 So amplification attack.
 So we started exploring whether we could do this to sensors and use these nation state
 sensors or even just beyond sensors, there's normal firewalls, like things that universities
 or just regular networked organizations have deployed.
 We discovered hundreds and hundreds, tens of thousands, millions of IP addresses that
 were behind these sensors that we could use to launch these attacks.
 And these attacks got crazy powerful.
 And the so the the who does it hurt more the sensors or the final recipients of the the
 attack?
 Yeah, so in this case, the weight is buried by both, but the brunt of the impact will
 be felt by the victim.
 Yeah, this line of work, it mucks with the sensor.
 But really, really, the some of the I want to say the purpose or something you can distill
 this work down to was sensors are causing more harm to the internet than they're not
 just the harm of a sensor is not just restricted to the citizens within its borders.
 Like a sensor anywhere is a threat to anyone everywhere.
 Yeah.
 So it's this work was less about let's flood a sensors network and more about let's prove
 to the world of these things are dangerous when they've been applied as carelessly as
 they've been deployed.
 Now other than block pages, you have some you have some very specific schemes of what
 you do specific to the censorship infrastructures that make these attacks even more powerful.
 What are examples of that?
 Yeah, so discovering these attacks in the first place, I'm making it sound very simple,
 right?
 You just send a request and then the response gets through.
 But I'm skipping over kind of an enormous step in here because what I've just described
 send a request pretending to be someone else should not be possible.
 Yeah, that that sentence should not exist.
 And it shouldn't be a thing you can do.
 And the reason that's the case is because when we make requests all the time, this happens
 I think there's a I think there's a gif in there that explains exactly what I'm saying.
 Just scroll up a little bit.
 There's a three way handshake that we need to complete.
 And that three way handshake is just this short exchange of packets.
 I think it's the one right above that.
 It's the short exchange of packets at the very beginning right here short exchange of
 packets that exists at the very beginning of our connection.
 And as an attacker, if I try and spoof a three way handshake, if I pretend to be my victim
 and start the handshake, the server is going to respond to the victim.
 And so I won't be able to get the critical bit of information I need from that handshake
 to finish it.
 And I need to finish that handshake in order to make a request.
 So throughout all of the all of networking history, basically up until this paper, it's
 been assumed that TCP, this underlying protocol behind all these requests is immune to these
 type of amplification attacks, largely immune.
 There's a small caveat there, but it's not worth getting into.
 So how do we go about addressing this problem?
 We used Geneva and AI techniques.
 And basically we replaced Geneva's fitness function and we told Geneva, hey, you can
 talk to these sensors, but instead of rewarding you for getting forbidden content, what we
 are going to do is we're going to reward you for getting content without establishing a
 connection and we're going to reward you for getting the biggest content you possibly can.
 So kind of turning the fuzz around its head a little bit and letting it explore the space
 of strategies that A, confuses the middle box into responding, so tricking it into thinking
 we have a connection already.
 And then B, once we've tricked it, getting the biggest possible response we can.
 And so this is a second set of work that was really powered by the same Geneva genetic
 algorithm.
 And we were able to use the same set of building blocks and primitives and programs that we
 had developed previously.
 We just applied them in a new way.
 And this is, if I understand it, it is not a weakness in TCP.
 Like if TCP were implemented correctly, Geneva wouldn't be able or shouldn't be able to find
 something around this, but this is specifically because these middle boxes are in there, right?
 Yeah, you're spot on.
 TCP itself is not the problem.
 It's the implementation of TCP.
 And that's partially why when we did this paper, we did this work, you can't just study
 TCP itself.
 You can't download the protocol specification, like think really hard, because that's not
 going to help you.
 We had to actually study real world sensors.
 So that's what we did.
 We took Geneva and we trained it against hundreds of sensors around the world.
 And then we took the results of that and were able to scan the whole internet.
 We scanned the internet almost 50 times actually, IPv4 internet, with these different packet
 sequences that Geneva discovered and effectively just attacked ourselves over and over and
 over again to see what kind of damage we could do.
 And how does that square?
 So before you said we're never going to release anything that helps the sensor in any way.
 And now you're releasing a recipe for launching massive attacks on something, right?
 I mean, I usually think any technology can be used for like with that, I could actually
 attack the sensor directly, right?
 And just make their life miserable using their own infrastructure, which is ironic even.
 I could use it to DDoS the Red Cross as well.
 So my perspective usually is that any technology can be used for good and for bad.
 But you've before said a little bit into the direction, we never want to publish anything
 that helps the sensor.
 This seems to be different.
 What's different here?
 Yes, the difference here is, and I want to note that we didn't just discover these and
 just immediately put them out into the world.
 We spent almost a year actually just doing responsible disclosure.
 We emailed every middle box manufacturer we could get in touch with and gave them advanced
 copies of our paper, advanced copies of this attack.
 We actually emailed, there's something called CERTs, Country Level Emergency Readiness Teams.
 These are teams that exist in various parts of the world that are basically designated
 to respond to network events pertaining to that region.
 So we emailed all of them around the world, so we were like, hey, that Chinese sensor
 you guys are operating, potential problem there.
 So we spent months and months working with DDoS manufacturers, CERTs, middle box manufacturers
 to try and patch these things and clean them up before this ever got out into the world.
 At the end of the day, this kind of runs into this broader responsible disclosure thing
 that a lot of the security field wrestles with of if I never publish this, there's often
 no incentive for this issue to be patched.
 Like if there's no downside to the network, they don't need to patch it.
 And if someone else discovers it before this gets out there, then they can start using
 it without the world and the defenders knowing about it.
 So there's this really tricky line you got to tow almost of I need to let everyone have
 as much time as possible to patch it, but they also need to know it's going to get out
 there to incentivize them to patch it.
 So with that in mind, we took the approach of let's take as long, as much time as we
 possibly can, let's tell everyone, any invested party about this attack, how to patch it,
 how to fix it.
 We gave them scripts to test their own network.
 And then after several months had passed and we were confident that they were, if they
 were going to take action, they already did, then we release the work.
 Cool.
 Yeah.
 Now you're a member of something that's called BreakerSpace.
 I've already mentioned it at the beginning.
 Do you want to maybe, because it's pretty unique, do you want to talk a little bit about
 what this is and what it does?
 Yeah, I'd be happy to.
 So BreakerSpace is a lab at the University of Maryland.
 Any UMD students watching, come check us out.
 The BreakerSpace lab, the kind of defining feature of this lab is that undergraduate
 students are invited to join and participate in the lab.
 So it's, the goal of this lab is to broaden and make research more accessible beyond just
 like PhD students and graduate students who are doing it.
 So this Geneva team and the broader censorship team within this lab has been staffed.
 I've been leading the team, but I've had a team of undergraduates who've been working
 with me on these projects.
 So every project we've talked about today and every paper on our website, this has not
 just been a one-man show.
 This has really taken a village to get these off the ground and get these moving.
 It's huge, huge tasks.
 And maybe you're missing, I didn't mention, a huge team of students who have been working
 on this with me.
 And okay, not unrelated to them being undergrads or not, did you, like how often does it happen
 that you get into like hot waters, like, you know, that there, you know, insecurity research,
 there are implicate, there are national defense implications, there are legal implications
 and so on.
 Like how do you navigate that space and how often does it happen that you're like, oops,
 I hope no one noticed this.
 It definitely, it definitely happens.
 And it's, we're really lucky to have such a supportive like university atmosphere in
 which we can do these things.
 We've worked closely with IRB, the Institution Review Board and our network security people.
 I mean, there was one week where we, for that scanning paper we were talking about, we're
 like, all right, let's kick off some scans.
 And then we immediately knocked out the university firewall.
 It's like, oh no.
 And they worked with us and helped us get it back and then helped work in such a way
 that wouldn't happen again.
 So what you're describing absolutely happens.
 I mean, one time we were accidentally, we didn't know this, we were accidentally attacking
 like the city of Jacksonville, Florida.
 And it was like, whoops, let's go email them.
 So that stops happening.
 Like the University of Kentucky, things like this.
 So what you're describing happens all the time.
 And it's like, oh shoot, whoops.
 And often those like whoops moments are like, that's a cool discovery you just made.
 We also got to go fix whatever you just broke.
 So totally happens, happens all the time.
 We've got lots of crazy stories like that.
 We're really lucky to have such a supportive atmosphere in which we can do these things.
 It's okay to break things as a work to fix them, obviously in such a supportive atmosphere.
 Where can people go if they want to get started in this space?
 Like let's say I'm an AI researcher.
 I want to have a good understanding of whatever reinforcement learning and evolutionary methods
 and genetic algorithms and all.
 But I've not much clue of security.
 Is there resources I can go to that you can recommend?
 So for security in general, there's so many, I mean, I'm sure there's two dozen YouTube
 channels that could probably hook you up with like incredible.
 So maybe we can send someone and link some of those below or something.
 I wish I could say that there is like this amazing AI censorship.
 I want to select censorship resource space where everyone can come to and learn how to
 apply AI to these techniques.
 Something like that doesn't quite exist, but there are great resources for learning about
 what censorship is happening in the world.
 So something like UNI.
 UNI is OONI.
 That's the Open Observatory of Network Interference.
 It's a spin out from the Tor team that monitors censorship all over the world.
 You can pull up the website later, but they can identify censorship in basically every
 country.
 It's run by volunteers and it's an incredible organization.
 So there's all sorts of groups like this that are studying censorship, monitoring for censorship.
 So for people who want to break into this more specific field of censorship, there's
 all sorts of great resources.
 Censored Planet is another group run by the University of Michigan.
 They're an awesome team.
 They also publish all their data.
 So all these groups have this very open sharing, hop on their website and they got lots of
 great resources, reports, data.
 You can get your hands in.
 Excellent.
 Is there anything else you want to get the word out to machine learning and AI people?
 Big open questions, anything that you feel should be out there?
 Especially just this whole space, this whole idea of there's this entire space of you can
 apply these techniques to in a way that's immediately impactful, helping real humans
 on the other side and humans who need this help.
 You have this potential to make a real immediate impact on the world.
 So it's a great space to get involved in.
 Excellent.
 Kevin, thank you so much for being here and bringing this a bit closer.
 I know more, I hope everyone else does too now.
 Thanks so much for having me.
 This has been a blast.
 Excellent.
 Super appreciate it.
 스포ated Adams How awesome was that?