How Apple scans your phone (and how to evade it) - NeuralHash CSAM Detection Algorithm Explained
neural networks artificial intelligence what is deep learning introduction to deep learning deep learning tutorial neuralhash neural hash apple privacy icloud privacy icloud encryption icloud illegal apple illegal apple scan apple scan illegal material icloud illegal material blinding step hash function private set intersection adversarial attack threshold secret sharing icloud csam csam apple csam apple scanning csam detection explained
#apple #icloud #privacy

Apple recently announced scanning all images uploaded to iCloud for CSAM (child abuse material), and that this scan would happen locally on users' phones. We take a look at the technical report and explore how the system works in detail, how it is designed to preserve user privacy, and what weak points it still has.

OUTLINE:
0:00 - Introduction
3:05 - System Requirements
9:15 - System Overview
14:00 - NeuralHash
20:45 - Private Set Intersection
31:15 - Threshold Secret Sharing
35:25 - Synthetic Match Vouchers
38:20 - Problem 1: Who controls the database?
42:40 - Problem 2: Adversarial Attacks
49:40 - Comments & Conclusion

Paper: https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf
ML News Episode about CSAM: https://youtu.be/gFkBqD2hbnU

Abstract:
CSAM Detection enables Apple to accurately identify and report iCloud users who store known Child Sexual Abuse Material (CSAM) in their iCloud Photos accounts. Apple servers flag accounts exceeding a threshold number of images that match a known database of CSAM image hashes so that Apple can provide relevant information to the National Center for Missing and Exploited Children (NCMEC). This process is secure, and is expressly designed to preserve user privacy.
CSAM Detection provides these privacy and security assurances:
• Apple does not learn anything about images that do not match the known CSAM database.
• Apple can’t access metadata or visual derivatives for matched CSAM images until a threshold of matches is exceeded for an iCloud Photos account.
• The risk of the system incorrectly flagging an account is extremely low. In addition, Apple manually reviews all reports made to NCMEC to ensure reporting accuracy.
• Users can’t access or view the database of known CSAM images.
• Users can’t identify which images were flagged as CSAM by the system.
For detailed information about the cryptographic protocol and security proofs that the CSAM Detection process uses, see The Apple PSI System.

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hello there. Today we're going to look at CSAM detection, the technical summary of Apple's system in order to detect child abuse material of users before they upload it to iCloud. So I recently reported on this in ML News and this story, of course, not my story, but the general story has sparked a lot of controversy around the world with respect to privacy of users and Apple essentially coming to users phones to scan the phones for illegal content and so on. So now we have the technical summary where Apple details exactly what's happening and how they're trying to both preserve user privacy, but at the same time, essentially catch people who create and share these types of materials. Now, needless to say, I think everyone's on board with reducing the spread of these materials. The question is what kind of trade-offs we're willing to accept in order to make that happen. And the trade-off here is mainly privacy of people, even though the system is designed to mitigate it, there are still weak points that where the system can be attacked, the system can be used for purposes that it was not intended. There are other problems. On top of that, at least in my estimation, the system can be evaded fairly easily. So, you know, you combine the system can be evaded fairly easily with we're going to implement the system that potentially has pretty, you know, really nefarious consequences if someone gets control of it that is not a good actor. I don't think you know, we'll have to think about the trade-offs of doing these types of things. And yeah, that's just that. So we'll go through the report, we'll go through how the system works, how Apple describes it. And we'll go through the strengths and weak points. And you can make up your own minds about that, even though I'm going to of course, try to bias you in a certain way. So keep that in mind. Alright, so we get here a essentially, it's a sort of a white technical white paper giving us a description, first an overview, and then a description of these various techniques. So there's going to be like a neural part with it, which is sort of the machine learning interface to this whole system. Since we're dealing with with images, that's, you know, that the front end, essentially, then we're going to deal with a whole bunch of cryptography slash security stuff, which tries to preserve user privacy as much as possible, while still allowing Apple to detect who shares this material. Okay, so here are the requirements of the system as far as Apple sees it. So first of all, the detection, so this is CSAM, it stands for child sexual abuse material. And the system specifically is designed to catch, identify and report iCloud users who store known material in their iCloud photos accounts. So it's very limited in scope. In fact, Apple does not scan your entire phone all the time for anything that you might have. It scans the things that you're about to upload to iCloud. And as we're going to, in fact, see it, it just computes as you upload to iCloud, it computes the security voucher and uploads that along with the material. And it only is supposed to detect known material. So there is a database, the database is provided by the National Center for Missing and Exploited Children. And that database, as far as I can tell, Apple doesn't even have necessarily access to that database itself. But for sure, they only so they they're not going to train a detector to, you know, classify abusive material per se, like, so they're not going to catch new material until that new material is entered into this database. So this is essentially saying we have a list, we have a big list, the database of things that we collected from, you know, confiscated phones or whatnot, collected from these websites. And we are simply going to check if in your iCloud account, there is any of those things, right? Any of any of those matches, then you have one of these known things, then we're going to report you. Now, the challenges, of course, to preserve user privacy. So here are the requirements that they set themselves to, they set upon themselves. Apple does not learn anything about images that do not match the known CSAM database. Now, this is hard, right? Apple can't just go to your iCloud account and and scan all the images. Otherwise, Apple would know what the other images are. So as I understand it, things in your iCloud are encrypted anyway, so Apple can't do that, right? So it can't just, you know, compare images, because otherwise, either you'd have to send the abusive images to the user's phone, which kind of defeats the purpose and then compare on the phone, or you have to send all the user's photos in clear text to the server. And then Apple would essentially see all the user's photos, which is also not okay. So we're going to have to get a bit creative here. Second, Apple cannot access metadata or visual derivatives for matched images until a threshold of matches is exceeded for an iCloud photos account. So it gets even more complicated, right? If you have apparently like if you have one image, they're not going to they don't want to, they don't want to report you yet, they are going to set a threshold, let's say five images, like if you have five matches in the database, then you know, it's very probable that you're engaged in actively sharing or consuming this material. And therefore, we're going to report you, you know, like if it's below that, probably their lawyers, their lawyers can't make a good enough case. And so they're going to say, if it's below a threshold, we don't want to be able to decrypt this, right? We only want to be able to decrypt all of the things once a threshold is exceeded. So this is yet an additional constraint that we have to somehow work with, we have to design an algorithm that allows us, we cannot decrypt anything until we have enough threshold exceedances. You know, excesses. Well, what's the word? I don't know. Okay, let's go through the other requirements more quickly a bit. The risk of the system incorrectly flagging an account is extremely low. In addition, Apple manually reviews all reports made to the to the to the Institute to the government to ensure ensure reporting accuracy. Now, this is a good goal, right? However, I think we've all encountered websites that told us that some decision was manually reviewed. But it's pretty, it was pretty clear that it wasn't right. So this is this is a goal, we know that as soon as there's like pressure, as soon as there is, you know, something more important going on, as soon as the system is overwhelmed, they are just going to swap out humans for for robots. I don't know how much pressure there needs to be for these humans to be swapped out. But still, at least initially, they're going to review all of the reports they make. Then users cannot access or view the database like this. Yeah, this should be fairly obvious. And users can't identify which images were flagged as being in the database by the system. So you can't design an algorithm that only transmits data to Apple once a match is found, because then the user would could inspect the network on their device. And they could figure out which of the which of the images is problematic, and apparently notify their whatever their friends or something. So you don't want that you want the users essentially to upload all their stuff, they never there's always a bit of data that goes with it. If there's a match, they don't initially know about it, I guess until the police knocks at their door. So these are the requirements. Okay. So this is a is an overview. What we have is we have this database of the database of this material, what we're going to do with this database is we're going to compute some hashes from it. So these are hash. Now a hash essentially is simply a representation of a piece of data that is shorter, but still uniquely identifies the data. So if I have a hash function H, and I input image A, I get out hash A. If I input image B, I should get out a different hash B. And if I input image A again, I should again get back back a okay, this is a classic hash, their hash functions are designed to if you if you input the same thing, you want to get the same thing out. If you input a different thing, you want to get a different thing out. And ideally, the thing on the right side, the hashes, they're much, much, much shorter, so much less data than the original data. This works because I mean, theoretically, it shouldn't work, right, but it works because most, most images that are possible in the data space aren't actually images. So the the amount of images that can exist as natural images is way lower than, you know, the pixel grid would allow. So there is a lot of compression potential. So the hash function is supposed to output the same thing. If you input the same thing, output the different thing, if you input a different thing, that's a classic hash function, we use hash functions when we want to check like the integrity of files. So in a classic hash function, if you change even one bit, the hash is going to change as well. That's how you see someone tempered with some some file or something like this. Here, we're going to use a little bit of a different kind of hashing, we also use these functions, but we also use this neural hash, which is going to be more fuzzy and geared towards the fact that we deal with natural data with natural images. In any case, what we're going to do is we're going to hash these hash functions from these images. And we're going to do a step that's called blinding, we'll look at that. And we put them on the client device. So the client device has the database, but in a hashed format. So looking at the hash will actually not tell you anything about the original image. So this is the requirement, the user does not see the images that are in the database. Okay, like that'd be terrible. In fact, okay, like the regular user doesn't see anything. But even if you inspect your device, you couldn't find that data because it's hashed. Now, on the client device, we take the image of the user, we, we compare it to the database. Now we can do that since the hash function output the same thing, if you input the same thing, right, if we run the image through the same hash function, if we run the image through the same hash function, we can simply compare with the database and see if there is something in the database that matches this image's hash. And then we know a hot that images in the database, it's a match. And then we can upload that to the cloud. However, that would violate another one of our requirements, namely, the user could learn which of the of their images match the database. So we'll have to, as I said, we'll have to get a bit creative. So what we do is we don't check for a match on the device, what we do is we produce this call so called safety voucher. The safety voucher is essentially comparing the image to the database, but it leaves out like one step in the process. And that step can only be done by the server. So so it's like a comparison, but you leave out the last step, it's actually not possible for the client device to do the last step of the comparison that would actually evaluate if something fits. And that's going to be done on the server. This technique is called private set intersection matching. And on the server, you do the matching if there is a match, you you know, you flash a red light, except there's the additional constraint that you need to have this threshold requirement. So you want that you can only decrypt the things of the user if a threshold is exceeded. And that is yet another technique called, I think threshold secret sharing or something like this. So we're going to look at these components one by one. First, the neural hash. Now, I told you about hash functions. And I'm going to repeat that the issue about a hash function is, if you input the same thing, it should output the same hash, it should output the same number. So here you can see an image on the top and the neural hash at the bottom. So this is the hash. So when we input the same image, we want the system to output exactly this number, not a similar number exactly this number. Now look at the image in the middle, would you say this is the same image or a different image? Now in the context of detecting abuse material, this is the same image, like it displays the same thing. We want our system to be robust to these transformations, because otherwise these people, they could just change the image a little bit. And then the hash changes, right, they could make it a little bit brighter or darker, they could just re encode it, they could resize it a little bit, and they would evade the detection. And that's what makes it difficult. What we can do is we can train neural networks to handle these kinds of things, we already have the techniques. So the two images you see here on the left, they should output the same neural hash. And the image here on the right, which is a different image, it should output a different neural hash. So what we're going to do is we're going to design a neural network in their case, it's a convolutional neural network, says it right here, a conv net, you input the image into a bunch of layers. And then at the end, you get out a vector. Okay, so you train this neural network, and you can do this via contrastive learning, this is essentially self supervised contrastive learning, such that if you input this image, and this image, their vectors are going to be fairly close together. And then if you input this image right here, its vector is going to be, you know, a lot different. So the vectors of images which are close in up to some transformations should be very, very close. This is standard self supervised learning, you teach the network to be robust to these kinds of transformations, you enforce that the vectors that the neural network outputs are close by each other, when you input these distorted images, and the network should also learn that images that are not distortions of each other, it should go far away. So we can do this, but you'll notice here the requirement is not fulfilled. Namely, they don't, the neural network doesn't output the exact same vector, it outputs only, we can only train it to output vectors that are really close by each other if it's a similar image, and really far apart, if it's a different one. So how do we get this discreetness in here, and that comes through locality sensitive hashing. So locality sensitive hashing is essentially a method in from from kind of the big data world to do approximate nearest neighbor search. And there is various techniques for doing this, I'm going to present you one of them, which I, from what I read, this is what they do, it might do something slightly different. But essentially, what you do is you define random hyperplanes. So one hyperplane might be this, and you know, in our case, it's just going to be a line, a 2d hyperplane. Sorry, a 1d hyperplane in a 2d space, one might be this, and one might be this. Okay, so those are your your three lines, let's number them. This is number one, this is number two, this is number three. And let's also label the sides of each. So this is the positive and the negative, positive and the negative, the positive and the negative side of that. So now what what can you do is you can check for each vector on which side of each of the three hyperplanes they are. So this vector right here, it would be on the positive side of plane one, it would be on the positive side of plane two and on a positive side of plane, three, so what this vector would actually be? You can even visually see they're in the same corner in the same slice of the space, whereas this vector right here, it would actually be on the positive side of plane one, and on the negative side of plane, two on the negative side of plane three. So here, you can see, it doesn't work for all vectors, work for all vectors, right, two vectors could be really close together, yet a plane could just cut through them. In that case, you would not find those two. But if you know, if you choose the number of planes correctly, their distribution correctly, then with very high likelihood, if you have two images that are very similar, and the neural network, in fact, outputs vectors that are close together for them, they will end up in the same bucket. So this here is going to be the discrete neural hash of that image. Now, they then stick that since this might still be a fairly high dimensional representation, depending on the hyper planes, they stick that into a classic hash function. So in order to reduce the number of bytes and also in order to make it less possible to in fact, reconstruct an image from the hash, because from these hashes, it's still actually possible to reconstruct the image, depending on the dimensionality, right? They feed that through more hash functions in order to to derive the neural hash. And there you see it. The neural hash for these two images, if we have trained the neural network correctly, should be the same in really like the same the same discrete bytes, whereas the neural hash for this image will be different. So that's how you detect and depending on how you train the network, you can catch most of these distortions, the network will also generalize. So even if some person comes up with like some transformation that you haven't specifically thought of, if you've done a good job at training, there's a good chance that you'll catch that transformation as well. So this is how we derive the neural hashes. Now, from the neural hash, so our first approach could be, you know, we take our big database of illegal material, right? So this isn't here is an image, here is an image, there's images, we run all of them through this exact same neural hash procedure, and we get a neural hash out of it. And then for a user, we take their image, we also run it through neural hash, right, that gives us some vector, and then we simply compare to the neural hashes of the database, which we have with us, this would work, okay. But as we said, this violates some of our requirements. Therefore, what do we do? So it's a bit more complicated. The server, the Apple has this database, or presumably they at least have these hashes, these ones of the database, right? What they're going to do is they hash them, they hash each of them one more time with let's call that H prime. So they hash each of them one more time with a hashing function that only they know, right? So they have the hashing function, it can also take like a private key. So there is a private key. And they call this the blinding step. Okay, so there's a hashing function that only Apple knows. Now, if your image if the user image goes here, they it gets like some sort of By the way, these lines, they are short for like, they're short for a vector of zeros and ones, right? So if I draw a line, it's like that's a it's a hash of an image. Now, if I have a hash of a user image, what I have to do is I have to send it to the server, because only the server has H prime, right? As this hashing function, and then the server can compare the two things. All right. So now this, so now this is, this is, this is better, this fulfills our requirements better. In order to also have the other requirements included, here is what we actually do. So what the server does is it derives the neural hash for each image in the database. And then it does this blinding step. Okay, so you receive a blinded hash from each image that the server knows that and then you order the things you order the hashes according to the neural hash. So how you how can you do that? You simply look at the neural hashes of each images and you put them in order, right? So yeah, you just sort them. So the order of the images is going to be according to the neural hash. So if I know the neural hash of an image, I can determine what row in the database it is stored at. However, the row is of course, a much shorter number than the neural hash itself. So I can't reconstruct the neural hash if I just from the row number. But I can if I have a neural hash, I can know what row in the database the blinded hash for that image is stored. So for the server, this essentially is double information, like this information comes from the image and this information also comes from the image. However, for the client, what the client now does is you get the client the device, you get the image, you compute the neural hash of the image. Now with the neural hash, you you do multiple things. So what you want to do is essentially you want to send the neural neural hash to the server, along with the payload. And the payload, just imagine it contains the real image, you put the real image into the payload, you upload that to the server, right, so the server can actually compare. But this would violate a bunch of our things. So what do you do? You take the neural hash, you look up the row, remember from the neural hash, you can look up which row it the blinded hash is stored at. Now, we have two cases, if the user image is an actual illegal image, right, then this blinded hash will be the actual blinded hash of this neural hash. So if I were to run this through H prime on the server, I would actually get the blinded hash. However, is the if the user image is not illegal material, you know, it will still have a neural hash, like you can compute that for any image, and it will still determine a row to look up because, you know, you'll get a row, you'll just probably get some random row. It's a it's a function that's only designed for the hashes that are in the database. So if you go to it with a hash that's not in the database, I'll just give you some row specifically, if you apply H prime to the neural hash, it will not output the same blinded hash. How can you now abuse this fact, such that the server cannot learn anything about your image if your image is in fact not illegal? Well, what you do is you look up you look up the row using the neural hash. And you use whatever is here in that row as a private key as an encryption key to encrypt the payload. And so you send you send the neural hash to the server, and you send the encrypted payload to the server. Here the payload, let's say the payload contains the actual clear text image. So we only want the server to be able to look at the image, if in fact, it's an illegal image. Again, let's play our two, is there a diagram? What happens on the server? No, let's play our two scenarios here. So the server gets this cryptographic header derived from the neural hash. The first thing it will do is it will run the neural hash through H prime, the server can do that, right? It will obtain it will obtain the blinded hash for that for that particular neural hash. Now, again, if in fact, this is an illegal image that should match this blinded hash right here. So it should be able the server should be able to decrypt the payload using that thing, right? Because it was, in fact, encrypted with this. So it should also be able to be possible to be decrypted with this, you actually don't need so this is only a conceptual thing, right? So this is what's happening. You take the neural hash, you compute the blinded hash for the neural hash, you can do that. And if you are able to decrypt the payload, that means that that the neural hash here actually resulted in this blinded hash here. Whereas if it was just kind of a random neural hash, the H prime will not give you the same blinded hash as is here as you used to encrypt. And therefore, you won't be able to decrypt the payload. Now, I was a bit hesitant when I when I saw this, because, you know, this is a this is a database, right? And the security here, you know, it's a good idea, but the security appears to rely on the size of that database, right? Because, sure, if this is like a giant database, you know, you have no chance of selecting the correct blinded hash from from here, like, all of this works. But let's say this is only like 100 rows, right? And we know the client used one of the blinded hashes in the database to encrypt their payload, like they had to they do this procedure where they look up the blinded hash, and they encrypt the payload with that. So there's a limited set of keys that the client could have used to encrypt the payload. So what keeps the server from simply trying all of them? I don't know that, honestly, like, I think we're just relying on the fact that this database is so large that the server can't try them all. But that means it must be something like exponentially large, which I don't think is happening. Maybe I'm missing something here. Maybe there is some additional thing. But I would guess, you know, if I'm Apple, and I really want to know what's in the payload, I just go through all of this database. And I just use all that because the key needs to be one of those things, right? Maybe I'm mistaken right here. But, you know, that's, I guess that's the thing. So this works, if you assume the server cannot just try all the blinded hashes, if you if you assume that, you know, the server, the only choice it has is to actually determine the blinded hash via H prime and try to decrypt, because only if in fact, this is the image that led to the creation of this blinded hash at this row in the first place, the this will actually match and the server will be able to decrypt otherwise not. Okay, so this is the first thing. This is the private set intersection, the client doesn't learn which objects matched, right, it just always uploads the neural hash and payload for every image. And the server is only able to decrypt if there was in fact a match and it learns nothing about the images for where there wasn't a match. So this this will fills our requirements. The next requirements is with respect to what's called threshold secret sharing. So this is private sec set intersection. The next thing that Apple wants is we don't they only want to know about you if you know if you've matched like five times or more. And that's, that's a technique called threshold secret sharing. And what we're going to do is we in fact are going to do two different levels of encryption. So remember, I said in this payload, there is the image, we put the image in there. This means if any of these matches the Apple gets to look at the image. So we're not going to do that. In fact, we're going to make it a little bit more complicated, we'll put like a little box into a box, you see this here, there's first encryption layer and second encryption layer. So the first encryption layer is going to be as we have it right now. But the second encryption layer is inside the first encryption layer. So even if there is a match and Apple can decrypt the payload and look at the payload, the payload itself won't help. And that is it's a pretty simple technique. In fact, there is a way in which you can create a key. So in I'm going to draw a key right here. A key in in cryptography, and you can shard it or make shares out of it. So what you can do is you can derive many, many shares as many as you want with the property that you can only decrypt whatever message I encrypt if you have at least let's say three of them. So if you have any three of those, then you'll be able to combine the three and in and decrypt the message that I encrypted, if you have less than three, then you're not able to. So we're going to encrypt. So inside this payload, we're going to encrypt the actual image information one more time with this key. And then for every payload we send, we only going to put one share of that key inside. So remember, whenever the neural hash of the image matches, which is up here, the server is able to decrypt this outer layer. So they will learn one share of the key. That means if you know, five of my images matched, the server was able to decrypt five of the shares. And then it has enough to decrypt all of the images. So repeat this box here. Repeat this box many times like one, two, let's do three, right? Repeat this box many times the cryptographic header up here, there is a box inside that can be decrypted when any of the ones match. And then inside there is a share of the key. And little box that you can only decrypt with the key with the payload inside of it. So once if if if only two things match, right, Apple doesn't have access to this in their box, let's say only to these two inner boxes, it cannot look at any of the images. But if three match, Apple has access to three of the inner boxes, which means it has three keys, and then it can go and decrypt not only the last one, but it can in fact decrypt all of the previous matches as well. So at that point, Apple will learn about all of the thus far encrypted payloads. So we have both Apple can never decrypt anything if the neural hash doesn't match. And Apple can only decrypt things when the neural hash match, neural hash matches whenever they enough matches have been made. There is a last thing in that. Yeah, so they display this in in various ways. There's a last thing in this. There's a last set here, where they generate synthetic match vouchers, because now, you know, let's say they can still see how many vouchers match, okay, so they do these synthetic vouchers in order to confuse themselves. So the devices will actually every now and then send dummy data. So they are called synthetic vouchers differ from real vouchers in the following ways. The underlying image information is substituted by dummy data. The secret chair of inner key is substituted by a random share that is totally independent of the inner encryption key. And the cryptographic header and the outer encryption key are chosen to always result in a match on the server. So you upload security vouchers that always result in a match. But the key share on the inside won't do anything because it's just like a random, random bit of numbers. So whenever you exceed the threshold, Apple will attempt to decrypt because it thinks it has enough shares. But if some of those things are synthetic shares, then it won't be able to. And this seems like this seems like a hurdle, this seems like it just makes introduces more noise. But this is exactly the goal, right? So Apple can never, if it just knows the number of matches, it says, well, we don't have enough matches yet to decrypt this person's account, it can never exactly tell how many matches of those are real. Because as long as they can decrypt anything, they have no idea if these vouchers are real or fake, right? And even if they like if they even if they have enough, like initially, before they have enough real ones, let's say this is a fake one, they can't tell which one is fake, they can only say, well, one of them is fake. Yeah, we need more. Okay, so there's, as you can see, there's a lot of mechanisms where the engineers here made deliberate choices to limit their own abilities, I'm going to guess they did this out of, you know, if you were, let's put that here. You know, if you're designing an algorithm like this, it's already hard enough to get the public to accept this. And they did, I think they did a pretty good job mitigating whatever they could, in order to say, look, here's how we're going to design it, we're going to maximally preserve user privacy in while still be able to do what we're doing. And this would all be good except, except this issue I mentioned here, you know, this would all be good weren't it for the pesky pesky deep learning. So where are the problems in the system as I see it? Where was this diagram here? So the problem in the system? No, here, the problem in the system are at the first of all, let's talk about this database. So you have a database that Apple presumably gets from this government institute. Well, sorry for scrolling around my devices. So presumably, Apple gets this thing from here, cool, you know, as long as that's the case, and as long as that database contains really images that are of child abuse, we're all we're all okay. However, this database is probably going to be quite guarded access to it is going to be limited. As I said, it's not even clear that Apple gets access to it. I mean, they, they probably do themselves a favor if they don't need access to it, they just send the neural network to the organization or to the to the government agency and say, please compute the neural hashes and send the hashes to us, we want nothing to do with this data whatsoever. That you know, Apple be smart doing that. That also means though, there are there's very tight control on that database. And not a lot of people are allowed to go and access the database. Good thing in principle, bad thing, if you think it in a different way, namely, what I can do is, I can, if I am the government, one of the few government officials that's actually allowed to interact with this database, I can insert a new thing. Now, if I'm a good, good bureaucrat, I'll insert new child abuse material because I want to find the people that share it. However, I can insert anything, right? And you know, there is an algorithm, if I insert something blinding step, yada, yada, yada, no one actually knows what's in the database, right? And then at the other end, it will some something will go bing, bing, bing, bing, bing, if that's actually on a phone of someone. So that this gives me as a government, this gives me a general mechanism, like I have to have to control Apple a little bit if Apple actually does the matching, but it's not even said it could be that Apple just forwards the decrypted information to the government. But you know, at the end, I have an algorithm, I insert anything into this database, any picture, but this is going to be this is this is just pictures is just the start, right? The they're going to widen this to all kinds of things. So I insert anything into the database. And you know, a second, a minute, an hour, a week later, I'm going to get big red lights for any single phone for any single iPhone that has that thing on their iCloud. This is the potential for abuse of this is enormous, right? If I'm a political party, I want to find my opposition, I just insert something into this database that I know is only likely on phones where my opposition is maybe I confiscated one of the phones and I just enter the stuff into the database. And then right after that, all the all the people that are part of the opposition of the rebellion of whatnot, light up and I know exactly who these people are. Right? So the Yeah, the potential for abuse for whoever controls the database is huge, because of the nature of the material, but also because it's a you know, a government agency, we are not going to be able to check whether the things in the database are actually what they claim they are. So Jen, like really big red flag for me there. Second of all, the image part, right in order to compute the neural hash on the device, and we saw this up here, this is computed on device, client device computes the neural hash of the image. Now, in order to do that, I need to have the neural network on my device. So I have an image here, I put it through the neural network, I get out a vector. Very standard neural network stuff. That's what that's what they do. They input stuff, they output vectors or whatnot. We there are things they're known as, as as adversarial attacks. And adversarial attacks can be run on technically any machine learning system. But it's really easy if you actually have access to the model, which you would if this is on your device, right. So what I can do with an adversarial attack is, I can remember when we said, even if two images are really close, they're only maybe you I crop them a little bit, the neural hash should be the same. This is true for, let's say random distortions distortions that happen naturally or anything you can think of. However, there are techniques called adversarial attacks, where you can specifically engineer the distortions such that the distortion to the image is minimal, like I only change a few pixels by a little bit, humans won't even notice it. But the output here will change drastically. Okay. So if I have access to the network and also have like if I have access to the LSH hyperplanes, it's really, really, really easy to create an adversarial attack that will switch the output just into a different bucket. This is this is insanely easy, right. And people that, okay, these might not be the smartest people that share this kind of stuff and upload them to iCloud. But one of them will come up with this idea and have a bit of a software engineering background. So if if you have a phone with root access, you could even, you know, install software that just automatically whatever picture you have, it automatically put some adversarial perturbation on it, such that the output is switched to a different bucket. As Apple says, if you if your image is legit, the probability that they'll they'll they'll match you is really small, which means most of these buckets are safe. So whatever you have to do, you just switch the bucket to some other bucket, you're going to be just fine. So it's quite easy to evade this, right? This is not like all this engineering afterwards, all of the private set in a crypto data, Ed, yada, Ed. This is all cool. But this relies on the fact that this neural hash is doing what it's advertised to do, which it is for normal images, but in the face of adversarial attacks, it is not. Now, there is a second thing in that I can if I can make two vectors be far apart when they should be close together, I can make two vectors be close together when they should be far apart, right? So if I have an image, and it would give me, let's say this vector, but I know this vector is a bad vector, right? This vector is illegal material vector, what I can technically do is I can make an adversarial perturbation that shifts this to that. And so that it ends up in the same bucket, while only changing the image a little bit. Now, this is a bit more complicated, because it requires me to actually obtain this bad vector, which I think the the general the way they hash everything, and so on, the only way of doing that is I would actually have to, I would have to obtain an image that I'm relatively sure is in one of these databases and then not get caught myself. And in order to derive this vector right here, which you know, don't like, this is this is an illegal step in itself, right? But if, if you're able to do that, then you're able to essentially frame people. So you can derive images that just look right, this this looks like I can take any image and do this, it looks like just a normal image, but it's perturbed in such a way that it matches with one of these illegal vectors, that'll be sent to Apple and so on. And now it depends if you really trust that everything here is manually reviewed or not. Yeah, again, the the potential here for for abuse is big. And if you now think of the fact that people who share this kind of material are probably going to employ some kind of these evasion techniques, like I presented here, some kind of these adversarial attack based evasion techniques, then, you know, it's the system is quite easy to evade. Yet, the potential for abuse, as we saw down here with, you know, who gets to do put what in the database, and the, I would say less less important but still present danger of people framing people, which also necessitates a failure of the manual review. Altogether, it the picture of whether this is a, a, you know, a desirable system to implement becomes less clear. So if I understood this correctly, I would be quite worried here. And I would like, you know, if I would like to see a world, I don't want to say I would advise I would not advise, but I would like to see a world where every single person in the world does does technique one right here to any image they have on their phone, right? It's like, if only one person uses encryption on the internet, like that's suspicious. But if everyone does it, you know, we're all, you know, it allows bad people to do bad things. Yes, because that's encrypted. But the ultimate safety for everyone is better. And you know, we'll have to look for other techniques to catch the, to catch the, the people sharing this this material. Yeah, so that that is kind of my, my, my take here. Yeah, I won't be doing this, though. I don't have iCloud. So yeah, hey, it's, it's going to be it's going to be interesting to see what's going to happen. In, you know, on top of all of this, in a general more meta, meta layer, we're about to see a step of where where the company essentially, you know, they don't scan every image on your phone, as I explained, but it goes into the direction of hey, you know, whatever you do with our stuff, we were going to essentially look at it, even if this algorithm we can't, but it is an expansion of the power of these companies, which is also worrisome by itself. Make of that as you will. This is already too long. Thanks so much for listening. If you like this, leave a like, subscribe. You know, if you have better ideas, I'm more than happy to read the comments here. If I got anything wrong, please tell me. Otherwise, have a nice day. Bye bye.
 Hello there. Today we're going to look at CSAM detection, the technical summary of Apple's
 system in order to detect child abuse material of users before they upload it to iCloud. So I
 recently reported on this in ML News and this story, of course, not my story, but the general
 story has sparked a lot of controversy around the world with respect to privacy of users and
 Apple essentially coming to users phones to scan the phones for illegal content and so on. So now
 we have the technical summary where Apple details exactly what's happening and how they're trying to
 both preserve user privacy, but at the same time, essentially catch people who create and share these
 types of materials. Now, needless to say, I think everyone's on board with reducing the spread of
 these materials. The question is what kind of trade-offs we're willing to accept in order to
 make that happen. And the trade-off here is mainly privacy of people, even though the system is
 designed to mitigate it, there are still weak points that where the system can be attacked,
 the system can be used for purposes that it was not intended. There are other problems. On top of
 that, at least in my estimation, the system can be evaded fairly easily. So, you know, you combine
 the system can be evaded fairly easily with we're going to implement the system that potentially
 has pretty, you know, really nefarious consequences if someone gets control of it that is not a good
 actor. I don't think you know, we'll have to think about the trade-offs of doing these types of
 things. And yeah, that's just that. So we'll go through the report, we'll go through how the
 system works, how Apple describes it. And we'll go through the strengths and weak points. And you
 can make up your own minds about that, even though I'm going to of course, try to bias you in a
 certain way. So keep that in mind. Alright, so we get here a essentially, it's a sort of a white
 technical white paper giving us a description, first an overview, and then a description of
 these various techniques. So there's going to be like a neural part with it, which is sort of the
 machine learning interface to this whole system. Since we're dealing with with images,
 that's, you know, that the front end, essentially, then we're going to deal with a whole bunch of
 cryptography slash security stuff, which tries to preserve user privacy as much as possible,
 while still allowing Apple to detect who shares this material. Okay, so here are the requirements
 of the system as far as Apple sees it. So first of all, the detection, so this is CSAM, it stands
 for child sexual abuse material. And the system specifically is designed to catch, identify and
 report iCloud users who store known material in their iCloud photos accounts. So it's very limited
 in scope. In fact, Apple does not scan your entire phone all the time for anything that you might
 have. It scans the things that you're about to upload to iCloud. And as we're going to, in fact,
 see it, it just computes as you upload to iCloud, it computes the security voucher and uploads that
 along with the material. And it only is supposed to detect known material. So there is a database,
 the database is provided by the National Center for Missing and Exploited Children. And that
 database, as far as I can tell, Apple doesn't even have necessarily access to that database
 itself. But for sure, they only so they they're not going to train a detector to, you know,
 classify abusive material per se, like, so they're not going to catch new material until that new
 material is entered into this database. So this is essentially saying we have a list, we have a big
 list, the database of things that we collected from, you know, confiscated phones or whatnot,
 collected from these websites. And we are simply going to check if in your iCloud account, there
 is any of those things, right? Any of any of those matches, then you have one of these known things,
 then we're going to report you. Now, the challenges, of course, to preserve user privacy. So here are
 the requirements that they set themselves to, they set upon themselves. Apple does not learn
 anything about images that do not match the known CSAM database. Now, this is hard, right? Apple can't
 just go to your iCloud account and and scan all the images. Otherwise, Apple would know what the
 other images are. So as I understand it, things in your iCloud are encrypted anyway, so Apple can't
 do that, right? So it can't just, you know, compare images, because otherwise, either you'd have to
 send the abusive images to the user's phone, which kind of defeats the purpose and then compare on
 the phone, or you have to send all the user's photos in clear text to the server. And then Apple
 would essentially see all the user's photos, which is also not okay. So we're going to have to get a
 bit creative here. Second, Apple cannot access metadata or visual derivatives for matched images
 until a threshold of matches is exceeded for an iCloud photos account. So it gets even more
 complicated, right? If you have apparently like if you have one image, they're not going to they
 don't want to, they don't want to report you yet, they are going to set a threshold, let's say five
 images, like if you have five matches in the database, then you know, it's very probable that
 you're engaged in actively sharing or consuming this material. And therefore, we're going to report
 you, you know, like if it's below that, probably their lawyers, their lawyers can't make a good
 enough case. And so they're going to say, if it's below a threshold, we don't want to be able to
 decrypt this, right? We only want to be able to decrypt all of the things once a threshold is
 exceeded. So this is yet an additional constraint that we have to somehow work with, we have to
 design an algorithm that allows us, we cannot decrypt anything until we have enough threshold
 exceedances. You know, excesses. Well, what's the word? I don't know. Okay, let's go through the
 other requirements more quickly a bit. The risk of the system incorrectly flagging an account is
 extremely low. In addition, Apple manually reviews all reports made to the to the to the Institute
 to the government to ensure ensure reporting accuracy. Now, this is a good goal, right?
 However, I think we've all encountered websites that told us that some decision was manually
 reviewed. But it's pretty, it was pretty clear that it wasn't right. So this is this is a goal, we
 know that as soon as there's like pressure, as soon as there is, you know, something more important
 going on, as soon as the system is overwhelmed, they are just going to swap out humans for for
 robots. I don't know how much pressure there needs to be for these humans to be swapped out. But
 still, at least initially, they're going to review all of the reports they make. Then users
 cannot access or view the database like this. Yeah, this should be fairly obvious. And users
 can't identify which images were flagged as being in the database by the system. So you can't
 design an algorithm that only transmits data to Apple once a match is found, because then the
 user would could inspect the network on their device. And they could figure out which of the
 which of the images is problematic, and apparently notify their whatever their friends or
 something. So you don't want that you want the users essentially to upload all their stuff, they
 never there's always a bit of data that goes with it. If there's a match, they don't initially know
 about it, I guess until the police knocks at their door. So these are the requirements. Okay. So
 this is a is an overview. What we have is we have this database of the database of this material,
 what we're going to do with this database is we're going to compute some hashes from it. So these
 are hash. Now a hash essentially is simply a representation of a piece of data that is
 shorter, but still uniquely identifies the data. So if I have a hash function H, and I input image
 A, I get out hash A. If I input image B, I should get out a different hash B. And if I input image
 A again, I should again get back back a okay, this is a classic hash, their hash functions are
 designed to if you if you input the same thing, you want to get the same thing out. If you input a
 different thing, you want to get a different thing out. And ideally, the thing on the right side,
 the hashes, they're much, much, much shorter, so much less data than the original data. This works
 because I mean, theoretically, it shouldn't work, right, but it works because most, most images that
 are possible in the data space aren't actually images. So the the amount of images that can exist
 as natural images is way lower than, you know, the pixel grid would allow. So there is a lot of
 compression potential. So the hash function is supposed to output the same thing. If you input
 the same thing, output the different thing, if you input a different thing, that's a classic hash
 function, we use hash functions when we want to check like the integrity of files. So in a classic
 hash function, if you change even one bit, the hash is going to change as well. That's how you
 see someone tempered with some some file or something like this. Here, we're going to use a
 little bit of a different kind of hashing, we also use these functions, but we also use this neural
 hash, which is going to be more fuzzy and geared towards the fact that we deal with natural data
 with natural images. In any case, what we're going to do is we're going to hash these hash functions
 from these images. And we're going to do a step that's called blinding, we'll look at that. And
 we put them on the client device. So the client device has the database, but in a hashed format.
 So looking at the hash will actually not tell you anything about the original image. So this is the
 requirement, the user does not see the images that are in the database. Okay, like that'd be
 terrible. In fact, okay, like the regular user doesn't see anything. But even if you inspect your
 device, you couldn't find that data because it's hashed. Now, on the client device, we take the
 image of the user, we, we compare it to the database. Now we can do that since the hash function
 output the same thing, if you input the same thing, right, if we run the image through the same hash
 function, if we run the image through the same hash function, we can simply compare with the database
 and see if there is something in the database that matches this image's hash. And then we know a hot
 that images in the database, it's a match. And then we can upload that to the cloud. However,
 that would violate another one of our requirements, namely, the user could learn which of the of their
 images match the database. So we'll have to, as I said, we'll have to get a bit creative. So what we
 do is we don't check for a match on the device, what we do is we produce this call so called safety
 voucher. The safety voucher is essentially comparing the image to the database, but it leaves
 out like one step in the process. And that step can only be done by the server. So so it's like a
 comparison, but you leave out the last step, it's actually not possible for the client device to do
 the last step of the comparison that would actually evaluate if something fits. And that's going to be
 done on the server. This technique is called private set intersection matching. And on the server,
 you do the matching if there is a match, you you know, you flash a red light, except there's the
 additional constraint that you need to have this threshold requirement. So you want that you can
 only decrypt the things of the user if a threshold is exceeded. And that is yet another technique
 called, I think threshold secret sharing or something like this. So we're going to look at
 these components one by one. First, the neural hash. Now, I told you about hash functions.
 And I'm going to repeat that the issue about a hash function is, if you input the same thing,
 it should output the same hash, it should output the same number. So here you can see an image
 on the top and the neural hash at the bottom. So this is the hash. So when we input the same image,
 we want the system to output exactly this number, not a similar number exactly this number. Now look
 at the image in the middle, would you say this is the same image or a different image? Now in the
 context of detecting abuse material, this is the same image, like it displays the same thing. We
 want our system to be robust to these transformations, because otherwise these people,
 they could just change the image a little bit. And then the hash changes, right, they could make it
 a little bit brighter or darker, they could just re encode it, they could resize it a little bit,
 and they would evade the detection. And that's what makes it difficult. What we can do is we can
 train neural networks to handle these kinds of things, we already have the techniques. So the
 two images you see here on the left, they should output the same neural hash. And the image here
 on the right, which is a different image, it should output a different neural hash. So what we're
 going to do is we're going to design a neural network in their case, it's a convolutional
 neural network, says it right here, a conv net, you input the image into a bunch of layers. And
 then at the end, you get out a vector. Okay, so you train this neural network, and you can do this
 via contrastive learning, this is essentially self supervised contrastive learning, such that
 if you input this image, and this image, their vectors are going to be fairly close together.
 And then if you input this image right here, its vector is going to be, you know, a lot different.
 So the vectors of images which are close in up to some transformations should be very, very close.
 This is standard self supervised learning, you teach the network to be robust to these kinds of
 transformations, you enforce that the vectors that the neural network outputs are close by each other,
 when you input these distorted images, and the network should also learn that images that are
 not distortions of each other, it should go far away. So we can do this, but you'll notice here
 the requirement is not fulfilled. Namely, they don't, the neural network doesn't output the
 exact same vector, it outputs only, we can only train it to output vectors that are really close
 by each other if it's a similar image, and really far apart, if it's a different one. So how do we
 get this discreetness in here, and that comes through locality sensitive hashing. So locality
 sensitive hashing is essentially a method in from from kind of the big data world to do approximate
 nearest neighbor search. And there is various techniques for doing this, I'm going to present
 you one of them, which I, from what I read, this is what they do, it might do something slightly
 different. But essentially, what you do is you define random hyperplanes. So one hyperplane might
 be this, and you know, in our case, it's just going to be a line, a 2d hyperplane. Sorry, a 1d
 hyperplane in a 2d space, one might be this, and one might be this. Okay, so those are your your
 three lines, let's number them. This is number one, this is number two, this is number three.
 And let's also label the sides of each. So this is the positive and the negative, positive and the
 negative, the positive and the negative side of that. So now what what can you do is you can check
 for each vector on which side of each of the three hyperplanes they are. So this vector right here,
 it would be on the positive side of plane one, it would be on the positive side of plane two and on
 a positive side of plane, three, so what this vector would actually be? You can even visually
 see they're in the same corner in the same slice of the space, whereas this vector right here,
 it would actually be on the positive side of plane one, and on the negative side of plane,
 two on the negative side of plane three. So here, you can see, it doesn't work for all vectors,
 work for all vectors, right, two vectors could be really close together, yet a plane could
 just cut through them. In that case, you would not find those two. But if you know, if you
 choose the number of planes correctly, their distribution correctly, then with very high
 likelihood, if you have two images that are very similar, and the neural network, in fact,
 outputs vectors that are close together for them, they will end up in the same bucket.
 So this here is going to be the discrete neural hash of that image. Now, they then stick that
 since this might still be a fairly high dimensional representation, depending on the hyper planes,
 they stick that into a classic hash function. So in order to reduce the number of bytes
 and also in order to make it less possible to in fact, reconstruct an image from the
 hash, because from these hashes, it's still actually possible to reconstruct the image,
 depending on the dimensionality, right? They feed that through more hash functions in order
 to to derive the neural hash. And there you see it. The neural hash for these two images,
 if we have trained the neural network correctly, should be the same in really like the same
 the same discrete bytes, whereas the neural hash for this image will be different. So
 that's how you detect and depending on how you train the network, you can catch most
 of these distortions, the network will also generalize. So even if some person comes up
 with like some transformation that you haven't specifically thought of, if you've done a
 good job at training, there's a good chance that you'll catch that transformation as well.
 So this is how we derive the neural hashes. Now, from the neural hash, so our first approach
 could be, you know, we take our big database of illegal material, right? So this isn't
 here is an image, here is an image, there's images, we run all of them through this exact
 same neural hash procedure, and we get a neural hash out of it. And then for a user, we take
 their image, we also run it through neural hash, right, that gives us some vector, and
 then we simply compare to the neural hashes of the database, which we have with us, this
 would work, okay. But as we said, this violates some of our requirements. Therefore, what
 do we do? So it's a bit more complicated. The server, the Apple has this database, or
 presumably they at least have these hashes, these ones of the database, right? What they're
 going to do is they hash them, they hash each of them one more time with let's call that
 H prime. So they hash each of them one more time with a hashing function that only they
 know, right? So they have the hashing function, it can also take like a private key. So there
 is a private key. And they call this the blinding step. Okay, so there's a hashing function
 that only Apple knows. Now, if your image if the user image goes here, they it gets
 like some sort of By the way, these lines, they are short for like, they're short for
 a vector of zeros and ones, right? So if I draw a line, it's like that's a it's a hash
 of an image. Now, if I have a hash of a user image, what I have to do is I have to send
 it to the server, because only the server has H prime, right? As this hashing function,
 and then the server can compare the two things. All right. So now this, so now this is, this
 is, this is better, this fulfills our requirements better. In order to also have the other requirements
 included, here is what we actually do. So what the server does is it derives the neural
 hash for each image in the database. And then it does this blinding step. Okay, so you receive
 a blinded hash from each image that the server knows that and then you order the things you
 order the hashes according to the neural hash. So how you how can you do that? You simply
 look at the neural hashes of each images and you put them in order, right? So yeah, you
 just sort them. So the order of the images is going to be according to the neural hash.
 So if I know the neural hash of an image, I can determine what row in the database it
 is stored at. However, the row is of course, a much shorter number than the neural hash
 itself. So I can't reconstruct the neural hash if I just from the row number. But I
 can if I have a neural hash, I can know what row in the database the blinded hash for that
 image is stored. So for the server, this essentially is double information, like this information
 comes from the image and this information also comes from the image. However, for the
 client, what the client now does is you get the client the device, you get the image,
 you compute the neural hash of the image. Now with the neural hash, you you do multiple
 things. So what you want to do is essentially you want to send the neural neural hash to
 the server, along with the payload. And the payload, just imagine it contains the real
 image, you put the real image into the payload, you upload that to the server, right, so the
 server can actually compare. But this would violate a bunch of our things. So what do
 you do? You take the neural hash, you look up the row, remember from the neural hash,
 you can look up which row it the blinded hash is stored at. Now, we have two cases, if the
 user image is an actual illegal image, right, then this blinded hash will be the actual
 blinded hash of this neural hash. So if I were to run this through H prime on the server,
 I would actually get the blinded hash. However, is the if the user image is not illegal material,
 you know, it will still have a neural hash, like you can compute that for any image, and
 it will still determine a row to look up because, you know, you'll get a row, you'll just probably
 get some random row. It's a it's a function that's only designed for the hashes that are
 in the database. So if you go to it with a hash that's not in the database, I'll just
 give you some row specifically, if you apply H prime to the neural hash, it will not output
 the same blinded hash. How can you now abuse this fact, such that the server cannot learn
 anything about your image if your image is in fact not illegal? Well, what you do is
 you look up you look up the row using the neural hash. And you use whatever is here
 in that row as a private key as an encryption key to encrypt the payload. And so you send
 you send the neural hash to the server, and you send the encrypted payload to the server.
 Here the payload, let's say the payload contains the actual clear text image. So we only want
 the server to be able to look at the image, if in fact, it's an illegal image. Again,
 let's play our two, is there a diagram? What happens on the server? No, let's play our
 two scenarios here. So the server gets this cryptographic header derived from the neural
 hash. The first thing it will do is it will run the neural hash through H prime, the server
 can do that, right? It will obtain it will obtain the blinded hash for that for that
 particular neural hash. Now, again, if in fact, this is an illegal image that should
 match this blinded hash right here. So it should be able the server should be able to
 decrypt the payload using that thing, right? Because it was, in fact, encrypted with this.
 So it should also be able to be possible to be decrypted with this, you actually don't
 need so this is only a conceptual thing, right? So this is what's happening. You take the
 neural hash, you compute the blinded hash for the neural hash, you can do that. And
 if you are able to decrypt the payload, that means that that the neural hash here actually
 resulted in this blinded hash here. Whereas if it was just kind of a random neural hash,
 the H prime will not give you the same blinded hash as is here as you used to encrypt. And
 therefore, you won't be able to decrypt the payload. Now, I was a bit hesitant when I
 when I saw this, because, you know, this is a this is a database, right? And the security
 here, you know, it's a good idea, but the security appears to rely on the size of that
 database, right? Because, sure, if this is like a giant database, you know, you have
 no chance of selecting the correct blinded hash from from here, like, all of this works.
 But let's say this is only like 100 rows, right? And we know the client used one of
 the blinded hashes in the database to encrypt their payload, like they had to they do this
 procedure where they look up the blinded hash, and they encrypt the payload with that. So
 there's a limited set of keys that the client could have used to encrypt the payload. So
 what keeps the server from simply trying all of them? I don't know that, honestly, like,
 I think we're just relying on the fact that this database is so large that the server
 can't try them all. But that means it must be something like exponentially large, which
 I don't think is happening. Maybe I'm missing something here. Maybe there is some additional
 thing. But I would guess, you know, if I'm Apple, and I really want to know what's in
 the payload, I just go through all of this database. And I just use all that because
 the key needs to be one of those things, right? Maybe I'm mistaken right here. But, you know,
 that's, I guess that's the thing. So this works, if you assume the server cannot just
 try all the blinded hashes, if you if you assume that, you know, the server, the only
 choice it has is to actually determine the blinded hash via H prime and try to decrypt,
 because only if in fact, this is the image that led to the creation of this blinded hash
 at this row in the first place, the this will actually match and the server will be able
 to decrypt otherwise not. Okay, so this is the first thing. This is the private set intersection,
 the client doesn't learn which objects matched, right, it just always uploads the neural hash
 and payload for every image. And the server is only able to decrypt if there was in fact
 a match and it learns nothing about the images for where there wasn't a match. So this this
 will fills our requirements. The next requirements is with respect to what's called threshold
 secret sharing. So this is private sec set intersection. The next thing that Apple wants
 is we don't they only want to know about you if you know if you've matched like five times
 or more. And that's, that's a technique called threshold secret sharing. And what we're going
 to do is we in fact are going to do two different levels of encryption. So remember, I said
 in this payload, there is the image, we put the image in there. This means if any of these
 matches the Apple gets to look at the image. So we're not going to do that. In fact, we're
 going to make it a little bit more complicated, we'll put like a little box into a box, you
 see this here, there's first encryption layer and second encryption layer. So the first
 encryption layer is going to be as we have it right now. But the second encryption layer
 is inside the first encryption layer. So even if there is a match and Apple can decrypt
 the payload and look at the payload, the payload itself won't help. And that is it's
 a pretty simple technique. In fact, there is a way in which you can create a key. So
 in I'm going to draw a key right here. A key in in cryptography, and you can shard it or
 make shares out of it. So what you can do is you can derive many, many shares as many
 as you want with the property that you can only decrypt whatever message I encrypt if
 you have at least let's say three of them. So if you have any three of those, then you'll
 be able to combine the three and in and decrypt the message that I encrypted, if you have
 less than three, then you're not able to. So we're going to encrypt. So inside this
 payload, we're going to encrypt the actual image information one more time with this
 key. And then for every payload we send, we only going to put one share of that key inside.
 So remember, whenever the neural hash of the image matches, which is up here, the server
 is able to decrypt this outer layer. So they will learn one share of the key. That means
 if you know, five of my images matched, the server was able to decrypt five of the shares.
 And then it has enough to decrypt all of the images. So repeat this box here. Repeat this
 box many times like one, two, let's do three, right? Repeat this box many times the cryptographic
 header up here, there is a box inside that can be decrypted when any of the ones match.
 And then inside there is a share of the key. And little box that you can only decrypt with
 the key with the payload inside of it. So once if if if only two things match, right,
 Apple doesn't have access to this in their box, let's say only to these two inner boxes,
 it cannot look at any of the images. But if three match, Apple has access to three of
 the inner boxes, which means it has three keys, and then it can go and decrypt not only
 the last one, but it can in fact decrypt all of the previous matches as well. So at that
 point, Apple will learn about all of the thus far encrypted payloads. So we have both Apple
 can never decrypt anything if the neural hash doesn't match. And Apple can only decrypt
 things when the neural hash match, neural hash matches whenever they enough matches
 have been made. There is a last thing in that. Yeah, so they display this in in various ways.
 There's a last thing in this. There's a last set here, where they generate synthetic match
 vouchers, because now, you know, let's say they can still see how many vouchers match,
 okay, so they do these synthetic vouchers in order to confuse themselves. So the devices
 will actually every now and then send dummy data. So they are called synthetic vouchers
 differ from real vouchers in the following ways. The underlying image information is
 substituted by dummy data. The secret chair of inner key is substituted by a random share
 that is totally independent of the inner encryption key. And the cryptographic header and the
 outer encryption key are chosen to always result in a match on the server. So you upload
 security vouchers that always result in a match. But the key share on the inside won't
 do anything because it's just like a random, random bit of numbers. So whenever you exceed
 the threshold, Apple will attempt to decrypt because it thinks it has enough shares. But
 if some of those things are synthetic shares, then it won't be able to. And this seems like
 this seems like a hurdle, this seems like it just makes introduces more noise. But this
 is exactly the goal, right? So Apple can never, if it just knows the number of matches, it
 says, well, we don't have enough matches yet to decrypt this person's account, it can never
 exactly tell how many matches of those are real. Because as long as they can decrypt
 anything, they have no idea if these vouchers are real or fake, right? And even if they
 like if they even if they have enough, like initially, before they have enough real ones,
 let's say this is a fake one, they can't tell which one is fake, they can only say, well,
 one of them is fake. Yeah, we need more. Okay, so there's, as you can see, there's a lot
 of mechanisms where the engineers here made deliberate choices to limit their own abilities,
 I'm going to guess they did this out of, you know, if you were, let's put that here. You
 know, if you're designing an algorithm like this, it's already hard enough to get the
 public to accept this. And they did, I think they did a pretty good job mitigating whatever
 they could, in order to say, look, here's how we're going to design it, we're going
 to maximally preserve user privacy in while still be able to do what we're doing. And
 this would all be good except, except this issue I mentioned here, you know, this would
 all be good weren't it for the pesky pesky deep learning. So where are the problems in
 the system as I see it? Where was this diagram here? So the problem in the system? No, here,
 the problem in the system are at the first of all, let's talk about this database. So
 you have a database that Apple presumably gets from this government institute. Well,
 sorry for scrolling around my devices. So presumably, Apple gets this thing from here,
 cool, you know, as long as that's the case, and as long as that database contains really
 images that are of child abuse, we're all we're all okay. However, this database is
 probably going to be quite guarded access to it is going to be limited. As I said, it's
 not even clear that Apple gets access to it. I mean, they, they probably do themselves
 a favor if they don't need access to it, they just send the neural network to the organization
 or to the to the government agency and say, please compute the neural hashes and send
 the hashes to us, we want nothing to do with this data whatsoever. That you know, Apple
 be smart doing that. That also means though, there are there's very tight control on that
 database. And not a lot of people are allowed to go and access the database. Good thing
 in principle, bad thing, if you think it in a different way, namely, what I can do is,
 I can, if I am the government, one of the few government officials that's actually allowed
 to interact with this database, I can insert a new thing. Now, if I'm a good, good bureaucrat,
 I'll insert new child abuse material because I want to find the people that share it. However,
 I can insert anything, right? And you know, there is an algorithm, if I insert something
 blinding step, yada, yada, yada, no one actually knows what's in the database, right? And then
 at the other end, it will some something will go bing, bing, bing, bing, bing, if that's
 actually on a phone of someone. So that this gives me as a government, this gives me a
 general mechanism, like I have to have to control Apple a little bit if Apple actually
 does the matching, but it's not even said it could be that Apple just forwards the decrypted
 information to the government. But you know, at the end, I have an algorithm, I insert
 anything into this database, any picture, but this is going to be this is this is just
 pictures is just the start, right? The they're going to widen this to all kinds of things.
 So I insert anything into the database. And you know, a second, a minute, an hour, a week
 later, I'm going to get big red lights for any single phone for any single iPhone that
 has that thing on their iCloud. This is the potential for abuse of this is enormous, right?
 If I'm a political party, I want to find my opposition, I just insert something into this
 database that I know is only likely on phones where my opposition is maybe I confiscated
 one of the phones and I just enter the stuff into the database. And then right after that,
 all the all the people that are part of the opposition of the rebellion of whatnot, light
 up and I know exactly who these people are. Right? So the Yeah, the potential for abuse
 for whoever controls the database is huge, because of the nature of the material, but
 also because it's a you know, a government agency, we are not going to be able to check
 whether the things in the database are actually what they claim they are. So Jen, like really
 big red flag for me there. Second of all, the image part, right in order to compute
 the neural hash on the device, and we saw this up here, this is computed on device,
 client device computes the neural hash of the image. Now, in order to do that, I need
 to have the neural network on my device. So I have an image here, I put it through the
 neural network, I get out a vector. Very standard neural network stuff. That's what that's what
 they do. They input stuff, they output vectors or whatnot. We there are things they're known
 as, as as adversarial attacks. And adversarial attacks can be run on technically any machine
 learning system. But it's really easy if you actually have access to the model, which you
 would if this is on your device, right. So what I can do with an adversarial attack is,
 I can remember when we said, even if two images are really close, they're only maybe you I
 crop them a little bit, the neural hash should be the same. This is true for, let's say random
 distortions distortions that happen naturally or anything you can think of. However, there
 are techniques called adversarial attacks, where you can specifically engineer the distortions
 such that the distortion to the image is minimal, like I only change a few pixels by a little
 bit, humans won't even notice it. But the output here will change drastically. Okay.
 So if I have access to the network and also have like if I have access to the LSH hyperplanes,
 it's really, really, really easy to create an adversarial attack that will switch the
 output just into a different bucket. This is this is insanely easy, right. And people
 that, okay, these might not be the smartest people that share this kind of stuff and upload
 them to iCloud. But one of them will come up with this idea and have a bit of a software
 engineering background. So if if you have a phone with root access, you could even,
 you know, install software that just automatically whatever picture you have, it automatically
 put some adversarial perturbation on it, such that the output is switched to a different
 bucket. As Apple says, if you if your image is legit, the probability that they'll they'll
 they'll match you is really small, which means most of these buckets are safe. So whatever
 you have to do, you just switch the bucket to some other bucket, you're going to be just
 fine. So it's quite easy to evade this, right? This is not like all this engineering afterwards,
 all of the private set in a crypto data, Ed, yada, Ed. This is all cool. But this relies
 on the fact that this neural hash is doing what it's advertised to do, which it is for
 normal images, but in the face of adversarial attacks, it is not. Now, there is a second
 thing in that I can if I can make two vectors be far apart when they should be close together,
 I can make two vectors be close together when they should be far apart, right? So if I have
 an image, and it would give me, let's say this vector, but I know this vector is a bad
 vector, right? This vector is illegal material vector, what I can technically do is I can
 make an adversarial perturbation that shifts this to that. And so that it ends up in the
 same bucket, while only changing the image a little bit. Now, this is a bit more complicated,
 because it requires me to actually obtain this bad vector, which I think the the general
 the way they hash everything, and so on, the only way of doing that is I would actually
 have to, I would have to obtain an image that I'm relatively sure is in one of these databases
 and then not get caught myself. And in order to derive this vector right here, which you
 know, don't like, this is this is an illegal step in itself, right? But if, if you're able
 to do that, then you're able to essentially frame people. So you can derive images that
 just look right, this this looks like I can take any image and do this, it looks like
 just a normal image, but it's perturbed in such a way that it matches with one of these
 illegal vectors, that'll be sent to Apple and so on. And now it depends if you really
 trust that everything here is manually reviewed or not. Yeah, again, the the potential here
 for for abuse is big. And if you now think of the fact that people who share this kind
 of material are probably going to employ some kind of these evasion techniques, like I presented
 here, some kind of these adversarial attack based evasion techniques, then, you know,
 it's the system is quite easy to evade. Yet, the potential for abuse, as we saw down here
 with, you know, who gets to do put what in the database, and the, I would say less less
 important but still present danger of people framing people, which also necessitates a
 failure of the manual review. Altogether, it the picture of whether this is a, a, you
 know, a desirable system to implement becomes less clear. So if I understood this correctly,
 I would be quite worried here. And I would like, you know, if I would like to see a world,
 I don't want to say I would advise I would not advise, but I would like to see a world
 where every single person in the world does does technique one right here to any image
 they have on their phone, right? It's like, if only one person uses encryption on the
 internet, like that's suspicious. But if everyone does it, you know, we're all, you know, it
 allows bad people to do bad things. Yes, because that's encrypted. But the ultimate safety
 for everyone is better. And you know, we'll have to look for other techniques to catch
 the, to catch the, the people sharing this this material. Yeah, so that that is kind
 of my, my, my take here. Yeah, I won't be doing this, though. I don't have iCloud. So
 yeah, hey, it's, it's going to be it's going to be interesting to see what's going to happen.
 In, you know, on top of all of this, in a general more meta, meta layer, we're about
 to see a step of where where the company essentially, you know, they don't scan every image on your
 phone, as I explained, but it goes into the direction of hey, you know, whatever you do
 with our stuff, we were going to essentially look at it, even if this algorithm we can't,
 but it is an expansion of the power of these companies, which is also worrisome by itself.
 Make of that as you will. This is already too long. Thanks so much for listening. If
 you like this, leave a like, subscribe. You know, if you have better ideas, I'm more than
 happy to read the comments here. If I got anything wrong, please tell me. Otherwise,
 have a nice day. Bye bye.