Extracting Training Data from Large Language Models (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper google apple openai berkeley stanford carlini dawn song google ai nlp natural language processing gpt gpt2 gpt-2 gpt3 gpt-3 gpt 2 gpt 3 bert transformers attention training data security leak privacy data protection ethics broader impact likelihood perplexity entropy url uuid personal information address private user data gdpr adversarial zlib
#ai #privacy #tech

This paper demonstrates a method to extract verbatim pieces of the training data from a trained language model. Moreover, some of the extracted pieces only appear a handful of times in the dataset. This points to serious security and privacy implications for models like GPT-3. The authors discuss the risks and propose mitigation strategies.

OUTLINE:
0:00 - Intro & Overview
9:15 - Personal Data Example
12:30 - Eidetic Memorization & Language Models
19:50 - Adversary's Objective & Outlier Data
24:45 - Ethical Hedging
26:55 - Two-Step Method Overview
28:20 - Perplexity Baseline
30:30 - Improvement via Perplexity Ratios
37:25 - Weights for Patterns & Weights for Memorization
43:40 - Analysis of Main Results
1:00:30 - Mitigation Strategies
1:01:40 - Conclusion & Comments

Paper: https://arxiv.org/abs/2012.07805

Abstract:
It has become common to publish large (billion parameter) language models that have been trained on private datasets. This paper demonstrates that in such settings, an adversary can perform a training data extraction attack to recover individual training examples by querying the language model.
We demonstrate our attack on GPT-2, a language model trained on scrapes of the public Internet, and are able to extract hundreds of verbatim text sequences from the model's training data. These extracted examples include (public) personally identifiable information (names, phone numbers, and email addresses), IRC conversations, code, and 128-bit UUIDs. Our attack is possible even though each of the above sequences are included in just one document in the training data.
We comprehensively evaluate our extraction attack to understand the factors that contribute to its success. For example, we find that larger models are more vulnerable than smaller models. We conclude by drawing lessons and discussing possible safeguards for training large language models.

Authors: Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, Colin Raffel

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi there. Today we're looking at extracting training data from large language models by what appears to be a big collaboration between corporations and academic institutions. There are almost as many affiliations here as their authors. So this is joint work between, you know, as you can see, many, many sort of institutions. And it is a pretty cool paper. So the high level topic is that these authors take large language models, as the title says right here, and train large language models specifically, and they're able to extract training data just from the trained model. In fact, just from the black box access to the trained model. And not only are they able to extract training data, they are able to extract pieces of training data, sort of verbatim, that have appeared only very few times in the training data. And they that's what they call a form of memorization. So they're able to extract these with a kind of pretty clever attack. So if you look at this prime example right here, they are able to query GPT two in this case, which is one of these large language models to output this piece of text. And the black stuff here is by the authors to protect the sort of privacy of this individual right here, this is though this is a real piece of text that they actually got out. And you can verify that. So they're able to extract this just from GPT two. And needless to say, this has consequences for security and privacy and so on. Because if you train one of these models with let's say internal or private data, user data, and so on, you have to be worried that these models are going to just output that data again, on the other end, and potentially leak information. This, of course, has not been a problem that much so far if you know, once we just trained image classifiers and so on. But here, especially with only black box access, this seems like it has some some consequences. So we'll go over the paper, we'll go over the the attack or the technique, the author's device, which is, I think, pretty clever. We'll go over sort of the results that they get from using this on a GPT two. And we'll go over my opinion of the paper, which I can already tell you, my ultimate opinion is that the attack is cool, the concerns are valid, but the paper is probably written a little bit more scary than it ultimately seems. In fact, I find the the results, the actual results of this paper fairly okay, like fairly promising, and sort of straightforward, not that scary. And also, the paper is interesting from another perspective, namely, from the perspective of what it tells us about these language models and how they work. And it it sort of strengthens a number of hypotheses that I've put forward in my video about GPT three, about how these models work. And that's also fairly cool to see in this paper. So we're going to jump in here. And as always, if you like content like this, don't hesitate to share it out, or subscribe and subscribe, I should say, if you're not yet. Alright, so they say it has become common to publish large, so billion parameter language models that have been trained on private data sets. This paper demonstrates that in such settings, an adversary can perform a training data extraction attack to recover individual training examples by querying the language model. Right, so we have a we already have quite a bit of information right here. So large language models have been, of course, trending with, you know, especially since GPT three, but at least since since the advent of the Transformers BERT and so on, though BERT isn't exactly a language model. So language models are models that, given a piece of text predict the next word, let's let's so easy as that or they predict the probability distribution over the next word. So if you say a cat sat on, so that's the input, the language model would give you a probability distribution over the next word. So the next word might be the or the next word might be a or the next word might be next, because of next two and so on. And it will sort of give you a probability distribution over each of these words that kind of looks like a face. It will tell you how likely each next word is and so on. And then you can sample from it, you can sort of choose one of those words and then go on. And you can evaluate the likelihood of entire sequences and so on. So GPT three is one of those large language models. And these large language models, they've been, of course, since they are large, we know that they also need a lot of data to be trained on. So a large language model would take like a giant piece, a database of training data, which is scraped from the internet usually. So this is too much to simply be curated by humans, they just let scrapers run over the internet, then they use this to train the model, whatever that is in GPT, GPT two in this case, and GPT two will then be a trained model. So you sort of throw the training data away, and you simply say, this is our model. Now, we're going to publish this, right. Now the problem is, if there is a piece of data in here, that is kind of secret. And you think, well, it's just one piece of data, like how much can how much can go wrong, right? The problem is, if I can inspect GPT two and recover this exact piece of training data, so that GPT two will output that exact piece, right, that is, is a problem. Now they make some good points here, this notion of a piece of training data, and what it means to memorize a piece of training data, and what it means to extract one is fairly fuzzy. And they go quite a bit deeper in this paper. So they have kind of strict definitions. They say, we demonstrate our attack on GPT two, a language model trained on scrapes scrapes of the public internet and are able to extract hundreds of verbatim text sequences from the models training data. These extracted examples include public personally identifiable informations, so names, phone numbers and email addresses, as you saw on the right here, IRC conversations, code 128 bit UUIDs, and so on. So they are able to extract all of these things from the trained model, right? And this, you can already see that how this can become a problem. They say our attack is possible, even though each of the above sequences are included in just one document in the training data. And this notion, this notion of memorization here, and when it is dangerous, they correctly say that this is only dangerous, of course, if the training example is contained in, let's say, only one piece of training data. Because if something is contained in thousands of pieces of training data, it's okay to memorize that, right? If a name of some famous person is memorized, and maybe the president of the USA lives at the White House, that it is not a secret, right? So it is okay if your language model remembers that, because it probably occurs in many training data points. However, if something is contained in just one document, right, and the model remembers it, then that is kind of true memorization. It is not maybe, or, you know, it's probably not learning anything from that data point, it's simply memorizing it to make its training loss lower. So that's the case on the right, right here. Though I have to say, this, as I said, it's written a bit more scary. So they don't exactly say that this name and phone number is contained in just one document. And they also say like, this is, of course, this is, this is on the public internet, GPT-2's training data was scraped from the public internet. So here is sort of my first investigation into this. First, you can Google this and you'll find it. You'll find this. And even though you know, the blacking out here also is a little bit of, I think it's a little bit gimmicky, because I don't see a problem with disclosing this particular piece of information. And I'll show you why. So when you search for it, you'll find the NIST homepage, you'll find a cryptographic algorithm validation program. And you'll find that this is a description of a software implementation. And here is the personally identifiable information, you can see, this is a corporate address. So this is a address of a corporation. And the contact information is a corporate contact is a corporate email address, it's a corporate phone number, and so on. This is the exact thing right here. And you know, with with respect to it only being present once in the training data. So if you actually search for if you complete the name here, and search for this, you'll find many, many, many, many, many results. Now, I don't know how many of these results are actually from, you know, in the GPT-2 training data, no one knows that, except OpenAI. So there's two Google pages of results. But oh, Google has D sort of D duplicated some of them. And now if I click on all, there are many there are 9000 results for this. And they are not all the same. Oh, no, no. So if you look at a bunch of those, you'll see that they are almost the same. But here, at the bottom, as you can see, this changes. So you know, depending on your scraper, these all count as separate websites. And therefore, I'm not so sure that this particular piece of information here is contained only once. Plus it is a corporate contact. So again, so to my point, the paper might be written a bit more scary than, than it ultimately turns out to be. Though, you know, you have to you have to make two different points like this particular piece of information. Yes, it might be written a bit more scary and gimmicky with the with the blacked out stuff. However, right? The paper has a point namely that if let's say you as a company do this on internal data, it might very well be. And they do have examples where they reproduce data from just one document. But even it might be that something like this happens to you internally, where you sort of maybe in your internal document base, you sort of do quasi duplicated document with the same information over and over. And and that's not the duplicated. And then your language model sort of memorizes that. So it's quite it, it has a point the paper. That's that's what I'm trying to say. I hope that's clear. Alright, so we'll get to the results in a bit. I hope I've already given you some sort of a taste for what you can expect. So first of all, they go into language models into sort of the definition of language models. And the language model here is simply framed as a model that can sort of give you a a probability of a sequence of text in sort of a stepwise fashion. So always probability of next word given the previous words, and you can evaluate that. Right, so the access to the model that they assume here is access to let's say the logits of the model or the output distribution of the model. And they say they use GPT two, because it's trained on large piece of text, but it's also you can you can evaluate it, it's not as slow, I guess as GPT three, and it's publicly available. However, the training data to GPT two is not publicly available. But they do have someone of open AI on the paper here. And this person at open AI made like mate, they could sort of query the open AI person to make sure a given piece of text that they find is or isn't in the training data of GPT two. So that's how they work. So that one per the open AI person acts as an API for the training data. Right, so they, they do, they define their attacks here. So they do a lot of things to, to set up cleanly what they do right here. So they have two points right here, there is this notion of memorization. Okay, so there's, they say there are many ways to define memorization in language modeling. In this particular piece of work, they say it is okay to memorize some stuff, they say language models must, for example, memorize the correct spelling of individual words, right, because the words are made of word pieces, and the language model needs to output that. So that's fine if it memorizes this. Indeed, there is an entire area of research that analyzes neural networks as repositories of memorized knowledge. For example, when GPT two is prompted to complete the sentence, my address is one main street San Francisco CA, it generates the next token 94107, a correct zip code for San Francisco in California. They say while this is clearly memorization in some abstract form, we aim to formalize our definition of memorization in order to restrict it to cases that we might consider unintended. So memorization as such isn't bad. What is bad is what they call here, the idetic memorization of text. So idetic memorization of text is when the model memorizes something that only appears very few times in the training data. So they say, we first define what it means for a model to X to have knowledge of a string. Our definition is loosely inspired. Yada yada yada, a model F knows a string, if s can be extracted by interacting with the model. So if you can input whatever you need to input, and the model outputs s, then the you say that model knows s, right. So if s is a piece of training data, then you say the model memorizes s, the model has memorized it. So here, they say a string is extractable from a language model if there is a prefix and the prefix here is the input to the model, such that if you input that model, the output will be the will be the string. And then they define this idetic memorization, respectively, they define k idetic memorization, a string s is k idetic, I have no clue whether I pronounce this correctly, k idetic memorized by a language model F, if F if s is extractable from F, so that's memorization, and s appears in at most k examples in the training data. Okay, so if this address of this person only appeared twice, but you could extract it verbatim from the language model, then that would be an example of two idetic memorization, okay, because k in that case would be two because it appears twice in the training data, though they they also, they are not clear what they mean by examples in the training data, because usually this training data is sort of chunked to make it fit into the language model and so on. And I think they do this on a document basis. So they would consider something like this here, one example, right, and then a different document, a different example. So if you have like, for example, if you have these IRC conversations that they are able to extract, so they claim here they are able to extract IRC conversations, or they're able to extract the user names of the IRC conversations, right? The user names might appear hundreds or thousands of time because they chat with each other. And it will all be, you know, in one document, but the document will be so long, they will actually be chunked into different training data pieces. Maybe I don't know. I don't know exactly what it means to be an example right here. But they do the example for sure, for sure, that piece of text can appear more than once, even if it is only in one example. In fact, they, they actually analyze the situation. Alright, so we've defined that this is the chi these k identity memorization. That's what we're looking for. That's sort of the problematic regime. If k is very small in the extreme k is one one piece of training data contains a string and we can extract the string at from the trained language model. They also say that for any given k memorizing longer strings is also intuitively more harmful than shorter ones. So this kind of makes sense. And they even they even go into sort of corner cases, they say that certain pathological corner cases, for example, many language model when prompting with the sequence, repeat the following sentence and then you give a sentence will do so correctly. This technically has any string to be known under our definition. But they, they of course don't do that. They assume they don't know the training data. So they can't just say repeat the following sentence and so on. But you do see that it is fairly hard actually to even define the problem right here, even though we as humans have a sort of an intuition what it means for a language model to unintentionally or un due to unintended memorization. Alright, so the adversaries objective here is to extract memorized training data from the model. The strength of the attack is measured by how private so how k idetic a particular example is stronger attacks extract more examples in total, and examples with lower values of k. They say we do not aim to extract targeted pieces of training data, but rather indiscriminately extract training data. While targeted attacks have the potential to be more adversarial harmful, our goal is to study the ability of language models to memorize data generally, not to create an attack that can be operationalized by real adversaries to target specific users. So you can see that here, they simply want some training data, they don't really care what it is, they simply want to get some so they're going to search for sort of the easiest to get training data. And that so they frame it as Yeah, we don't want to devise an attack that can attack individual users. But there is a different component to it. So if you had to sort of guess the password of any particular user, that would be you know, fairly, fairly hard. However, if you had to guess a password that was used by any user, it's fairly easy, right? Even if you discard the fact that most of people use password as password, and so on, if if people would just uniformly sample words from the dictionary as their password, still you'd have a decent chance of figuring out a password, right? We have a decent chance of figuring out, you know, not super high entropy things like maybe credit cards, you'd have a decent chance of figuring out the credit card number, just by guessing one. So this is the regime we are in here. And it's entirely different regime, I think, if you try to attack individual users. Essentially, what they're going to do right here is they're going to say, look, there's training data, right here. Now, some training data, these models can extract a pattern from, right? If and this is what we do with machine learning, right? We say, okay, this this data right here, they all have like some pattern. And this data right here is some pattern. And you can learn from this. And it has some pattern. So the machine learns to sort of abstract from extending data samples, and so on. But here is a data point that doesn't really fall into any of these categories. So what the model will do is it will simply say, well, this is its sort of own little group, I'll remember that I can extract some pattern from here and from here, but I can't extract any pattern from here. But I need to get my loss down. So I'll just remember that, you know, individual piece of training data. And that's exactly what we can recover with this sort of attacks these individual pieces that aren't really don't really have anything close, there is not really a pattern to it. So the best the model can do is remember that it doesn't mean that with this attack, you're going to get this piece of data or this piece of data, right. So if your personal identifiable information is sort of falls into some kind of regular pattern, it's, it's likely to be more safe against an attack like this. That's why they, for example, are able to extract these sort of UUIDs, or URLs with random strings in them, because random strings have no pattern, right. So they are likely to be out here away from the other training examples, where the best the model can do is actually remember the thing, rather than extract a pattern. Now, the other example here with this personally identifiable information, I believe that's just because it appears a lot of times, honestly, not because there is no pattern, but because it appears so many times that the model simply, you know, it's, it's, why should it extract a pattern when it appears so often, it can just, you know, remember it like a famous person's name seems to be an address that's important if it appears so often, I guess, from the point of view of the model. So that's, that's sort of what this does, again, it extracts indiscriminately, it doesn't mean that the attack can be leveraged to, you know, get any training data sample back. It's still worrisome, but you have to take into account. Another thing that that is really sticking out in this paper is the amount of hedging that this paper does. This, this almost in every paragraph, but certainly in every subsection, there is like hedging, hedging against, you know, why it is okay to publish this research, and so on. So, you know, when they say our attack target is, is GPT two, we select GPT two is a nearly perfect target from an ethical standpoint, the model and the data are public. So any memorized data we extract is already public, and so on. And they do this in in every piece of text. And, you know, in my video about broader impact statements, that was exactly my my point, these large corporations, right? If many, many of these authors, I think a fair amount of work went into framing this research, such that it sort of can't get attacked from, you know, people concerned about, you know, ethical considerations when releasing research like this, like this is clearly research that can be leveraged, you know, for for bad, if you will. But since these, you know, companies have a lot of resources, and, and there, you know, can put many people on this can devote fair bit of amount of of work into framing the problem that can be mitigated. Whereas if you know, some lonely PhD student would do the same research right here, the exact same research, I very doubtful it would be received as well as this piece right here. And in my opinion, as I already said in that video, this just sort of shifts, you know, a bit more power to these large institutions that sort of can afford the framing right here, they don't have to change anything about their research. But the rest of us do. All right, rant over. Let's continue. So they, they're going to do this in two different steps right here. And they have a diagram. Yes, I have a diagram. So first, they do this in two steps. Step one, they query the model, they have different queries, right, but they just sort of generate data from the model. So they generate lots of data right here, from the model. Then they select somehow they select from the model, a subset that they think these could be memorized training examples, then they do the duplicated, they they select again, and then they check, okay, this is it's fairly, fairly easy workflow. So step one is generate a bunch of data that you think could be memorized. And then step two, check whether you find these samples in the internet, because all of GPT two is training data comes from the internet. If you can find them on the internet verbatim, right, that probably means GPT two as remember, like the likelihood that it verbatim remembers, you know, I UUID that wasn't in its training data is almost zero. So yeah, this this goes by manual internet search. So respect to these authors who have done this, they start out with some fairly, fairly weak baseline, which is they simply generate the large quantity of data by unconditionally sampling. And then they predict which output contains memorized text by simply analyzing the likelihood. So whatever text the model finds highly likely, they they think that could be memorized. Because if you provide a model with training data, and you ask it to reduce its loss on the training data, it will assign highest likelihood to the training data. That's, you know, just, that's how these models work. So they assume that if a model has high likelihood, or low perplexity, that's the sort of same thing. Yeah, so you can see here, if the perplexity is low, then the model is not very surprised by the sequence and has assigned, on average, a high probability to each subsequent token in the sequence. And if that happens, they say, this could be memorized. This is obviously, obviously, very, very, very simple. See, this simple baseline extraction attack can find a wide variety of memorized content. For example, GPT-2 memorizes the entire text of the MIT public license, as well as the user guidelines of Vaughan life, an online streaming site. While this is memorization, it is only k-idetic memorization for a large value of k. These licenses occur thousands of times. The most interesting examples include the memorization of popular individuals, Twitter handles or email addresses. In fact, all memorized content we identify in this baseline setting is likely to have appeared in the training data set many times. So here they say it doesn't really work if you just sample and then look at what's most likely because yes, this will be memorized, but it is sort of a non problematic form of memorization like famous people's Twitter handles. That's this is like famous people's names at this point, right? So now they go about improving it. Okay, so they improve both steps, they improve step one. Where are we? No, it's down here. They improve step one by doing one of two things. Either you want your temperature to decay. So in this sampling, when you sample from the model, you have a temperature that you sample from, and you can decrease that over time. So at the beginning, you can let the model explore a bit, but then you can you can decrease it. And that's so the sorry, the the goal of changing step one is to create a more diverse set of generations, right? So you can sample with high temperature at the beginning, and then decrease it over time. Okay, such such that you still get sort of high likelihood sequences, but you get different ones. So you start off differently, and then you go into the high likelihood regime. The second way they change this is what they do is they go to the internet again. So they go to the World Wide Web, which is okay. I'm terrible at drawing the globe with okay, they go to the World Wide Web. And they just get pieces of text from the internet. So they get a website. And they just take some tiny substring from here, from this, and they use that as the input to their model. And that's sort of to get more diverse predictions. So if you input a short prefix that you found somewhere on the internet, and then let the model continue, that generates you have wide diverse variety of pieces of text. Okay. So that's how they up the DA how many different samples the model generates. Because in the initial experiments, they found that the model will sort of output the same things over and over again, if you simply query it unconditionally. So either high temperature or conditioned on internet text. The second step is sort of what I find the clever step. So here, they have to before they simply said, whatever has high likelihood, that's what we think is memorized. But of course, a lot of these will not be, you know, with low K memorized, a lot of them will simply be high likelihood because they're actually likely. So they say, okay, when, when is when are we in this situation? So let's say here is the here is our data set, okay. And here is the the MIT public licenses here. And it you know, it appears like billion, billion, billion times. So this data point is like ginormous. It's all you know, the MIT public license. And here is our outlier data point. Now, this model will extract patterns, let's say from this, and this is a pattern. And it will assign a single pattern to the MIT public license, because it just appears so often. And it will assign a single pattern to this data point down here, just because it's such an outlier, right? So how do we how do we devise a scheme that will find this one reliably, but sort of will recognize, wait a minute, this this memorization here is okay. But we need to devise a scheme without having access to the training data, right? If a human looks at it, of course, the MIT public licenses, oh, seems common, we know that it's common, and so on, we know that it's highly likely text, because it's a, it's a license almost everywhere. If a human looks at this right here and sees, you know, the name and address of a person or a credit card number, we know that's not really highly likely text. And that's sort of the answer right here. So we say if a human looks at it, but what is a human, a human is just another language model, among other things, right, but the human is just sort of another thing that has an intuition of how how likely text is. So the basis of their approach is going to be the following. Let's take a second, second data set, okay, sampled in the same way also from the internet, but not in exactly the same way. In fact, they use common crawl instead of the the Reddit outbound links that GPT two used. But we take any other data set, and I'm going to draw the other data set. So here's the data point, here's the data point, maybe this data point is duplicated from the other data set. And here's the data point here one, right, so you're going to have sort of other data points. But also, you know, since you're sampling from the internet broadly, you're going to have the MIT public license many times. And you're also going to have the outliers in this data set. Now, the important part is, you're probably if you sample this differently, in the same fashion, but a bit differently, you're probably not going to have this same outlier right here, you're probably not going to have that in your new data set. Okay, so you can see in the new data set, I hope you can see this, you're going to have the the same pattern extracted here, even though it's from you know, slightly different data points, you're going to have maybe a pattern extracted here, maybe one here, you're going to have this same cluster here, because the MIT public license will appear, even though it comes from other documents, it's copied over and you're going to have this outlier right here. So what you can do to differentiate our two our two things, you can consider a second language model. And you can ask, so here you have two things that the first language model things are very likely, you have this thing right here. And you have this thing right here, both the first language model consider super likely, you ask the second language model, and the second language model says, yes, the MIT public license, I consider that to be also super likely. But this outlier over here now that's I've never seen that what's that that seems very unlikely. And so by the ratio of the two likelihoods of the two different models, you can find out samples that the first model finds super likely, but the second model things are not likely at all. And that's exactly the trick they use right here. In fact, they use many instances of that trick. So here are the strategies perplexity is simply what they use before whatever is likely is probably memorized. This yes, it's memorized, but it's often memorized justifiably. Then they have these strategies small and medium. And and this is the ratio of the log perplexities of the largest GPT two model, that's the one they attack and the small GPT two model. And this ties into so you don't even need a different model, right, you can simply train a the reason they train a smaller model is the following. And we on the machine learning street talk podcast, if you don't know that it's a it's a podcast where we talk to people from various, you know, from the industry in from various research labs, and so on. And we spoke with Sarah Hooker, who we talked about their paper, the hardware lottery, but she also has other research, where she sort of shows that if you have weights, so you have a neural network, and it has, you know, layers, layers, layers, and you have weights in these layers, right? What she was able to show is that not all weights are equal. So some of the weights, let's say the weights here will be allocated to these pattern extraction things. So you know, here we have these, you know, you have data training data training data outlier outlier, right? So you'll have this, you have these weights representing this pattern within a layer, right? You have these, this pattern will be represented by these weights right here. And then you'll have other weights, they're sort of allocated to remembering single or very few outliers. Okay, so here, this will be allocated. And these will be disproportionate. So there will be many, many more data samples covered by, let's say, this piece of weights right here, I should have drawn the bottom one smaller than by this. So there might be, you know, 1000 training examples, covered by one piece of weight space. And there might be only one piece of training data covered by this other piece of weight space. And that's simply because it can extract a pattern from one but not from the other. So it needs to memorize it. And the larger we make these models, you know, the more parameters we give them, the more the more the more ability they have, the more space they have to do this remembering. So what what Sarah Hooker noticed in her paper is if you then distill these models, and distillation is the process of taking these models, and putting their knowledge into smaller models, then what happens is not all training data points will will so that in distillation, you usually lose performance, not all training data points will lose performance equally, namely, you will lose performance on the training data points that are sort of these outliers that are these not often represented in the training data that you know, the model has a harder time extracting patterns from it. So they will be seldom patterns, or just hard patterns, I would also assume that, you know, patterns that are harder to extract will also fall, fall away. So the the more complicated patterns will also be sacrificed. But I guess, among the things are these outliers. So if you train a smaller model, the smaller model would have less ability to remember these outliers. And therefore, if you do this, you don't even have to do it on a different training data set, right? You can simply compare to the same model trained on a sorry to a smaller version of the same model trained on the same training data set, because that will probably not remember the outliers as much. It would have been interesting if these authors here had actually distilled GPT to and though they do not have access to the original training data, so I can get why they didn't do it. But would be interesting to see that. That gives me an idea sort of maybe there is actually a way to look at the weights and I get these these authors don't have access to the weights, but maybe there's a way to look at the weights and to actually be able to sort of in some way spot right which of the which of the weights only are associated with with single or very few training data points. Maybe during training, you can sort of count how many times a weight is updated in a substantial amount or maybe looking at the attention matrices, you can sort of determine what are the kind of patterns that need to happen that lead to this weight being activated, right? So if there is a weight, and it's activated by lots of lots of different patterns, maybe, you know, that weight is useful for many, many forward propagated signals. But if there is another weight that's only activated by a specific pattern, right, then maybe that's one of these these memorization weights. So maybe there's a way to recognize these in the weights directly. So distillation appears to be sort of a defense against this this memorization of things, though that's not that's not done in this particular paper, they also have different strategies. So you don't need to do this neurally, right, you can compare the ratio of the perplexity that GPT two gives to the zlib entropy. So this is simply a text compression method, you can even compare it perplexities between the original string and the lowercase version, and so on. So they extract for each of these configurations, we select 100 examples among the top 1000 samples. So they produce 1000 samples, and they sample 100 from those 1000. So they mostly sample from low ranked samples, but also they explore some of the high ranked samples, they have a formula where they sample, they de duplicate, and then they investigate. All right, so they do Google searches, if they can find the thing, they say that's memorized. Alright, so they say, across all strategies, we identify 604 unique memorized training examples from among the 1800 candidates, our best variant has a true positive rate of 67%. That's quite remarkable, right? So 67% 67% of the things that this method delivers you automatically are actually memorized. Though you have to qualify that right? If you want more than 1000 examples, that rate is going to drop, right? You since you select the top 1000 examples, these are the most likely to be memorized. So yeah, if an attacker wants more, if they want to scale this attack up, their positive rate is gonna plummet fairly quickly, I'm going to assume it would actually be interesting also to see how that develops with the top the top retrieve document right here. But I get the they have to do Google searches to figure out and then ask open AI to figure out if if it's really a memorized training example. They say their categories, we manually group the memorized samples into different categories. The results are shown in table one, most memorized content is fairly canonical text from news headlines, log files entry from forums or wikis or religious text. However, we also identify a significant amount of unique data containing 128 bits UU IDs correctly resolving URLs containing random strings and contact information of individual people. Okay, so as I said, these, this is this is fairly interesting, but also a bit expected, right? If I give you the start of a UUID, then there is no pattern to extract, except I guess the UUID structure, but there is no deeper pattern to exact. So all the model really can do is memorize the UUID, especially if there aren't too many UUIDs in the training data, or if this particular UUID is some sort of, as I said, it's this outlier type of situations, the same thing for, you know, URLs containing random strings, these are just not pattern extractable, therefore, easily, more easily remembered by the model than learned. So you can see right here, the breakdown, where they see how many of what they extract. And here contact info, 32 named individuals non in non news 46. That's a fair amount of things you can extract from GPT two, you have to say that that is all right, all of GPT two, you get approximately 100 things that are kind of names or contact informations. So as I said, not too bad, specifically considering what I've shown you here, right? They, that's one of these contact informations. And they do say this in the paper that this person, this information was obviously released in the context of this software project. And the problem is only the model might actually output this in a different context, right? The model might think, oh, now I need to output some sort of name and address, what kind of names and addresses to know what this name and address appears pretty often, I'm going to put that here. And so that's a failure case, you know, that these things can do. So here is a sort of a graph. And they have more of these graphs later. But you can see that here, for example, is a GPT two perplexity. And here is this z lib entropy. And if you plot them one against another, most things will fall on this diagonal right here with, you know, the giant blob around here for most texts of the internet. And there will be a region where GPT two things, this is fairly low perplexity. But z lib thinks the text is relatively high entropy. So these are candidates for memorization. And the red and blue here are the ones the authors selected for checking. And the ones that are blue are ones that they found are memorized from the internet. So a fairly high percentage, in fact, 67% of this method that they selected was, in fact, was memorized. Though, as I said, you can see that there aren't super many more, right? So this is this is all samples. And I don't know how many, you know, they could generate more, but you can see that it gets pretty sparse out here. Okay. Yeah, so examples of memorized content, personally identifiable information. They say there are several examples of individual people's names, phone numbers, addresses, and social media accounts. Some of this is memorized content is just exclusive to a few documents. For example, we extract the usernames of six users participating in an IRC conversation that happened in exactly one document. Yeah. So I guess the question is, how often did the usernames appear in that one document, right? And once the model sort of, and how, how distinct are these usernames from other usernames? Because if they're very distinct, and they happen, you know, they have a long conversation, it can be easy to see that the model will remember that not saying this is not a problem. I am telling you, the models, it's not, it's not that they'll just randomly remember stuff, there needs to be very specific conditions for the models to remember stuff. So they say, we identify 50 examples of memorized URLs that correctly resolve to live web pages. Okay, many of these URLs contain uncommon pieces of text, such as random numbers or base 64 encoded strings. Again, this this random element right here makes it you can't extract a pattern. They say we identify 31 generated samples that contain snippets of memorized source code. And they can actually extend that. So they can take these snippets and they always, I think they do 256 token length, but they can extend that to sort of verbatim recover the source code. And that's also you know, that's that's fairly interesting. And unnatural text, yeah, these UUIDs. A Google search for this string identifies just three document containing this UUID. And it is contained in just one GPT-2 training document. Okay, though, again, we are not seeing how often they say table three gives nine examples of k equals one memorized content, each of which is a random sequence between 10 and 87 characters long. You can see the table right here. So these are examples of random strings that for some reason appear in this training data in exactly one document. However, this string right here, for example, appears 10 times. And this string right here appears 311 times. So again, it's a random string that appears 10 times is fairly often for a piece of text to appear, especially the same piece of text that is not pattern close to any other piece of text. It seems okay that the model remembers that it seems expected, right? So yeah, here, they also say data from two sources, we find that samples that contain two or more snippets of memorized texts that are unrelated to one another. In one example, GPT-2 generates a news article about the real murder of a woman in 2013, but then attributes the murder to one of the victims of a nightclub shooting in Orlando in 2016. And this I found very, very interesting, right? Because that's exactly what I said GPT-3 does, right? Especially so in GPT-3, they have this example of GPT-3 writing an entire news article about, I'm not even sure about some pastors, some split in the Mormon church or something like this, or I don't remember correctly, but I was able to Google that. And I did not find the verbatim sequence, but I found that article that GPT-3 wrote many, many times in sort of different words in written down in, you know, books and reported about and so on. So what GPT-3 did is simply, I would guess interpolated between these things. And here they find the same thing GPT-2 just takes two pieces of text and sort of finds that they're close and sort of interpolates between the two. I would call this memorization two and they say, yeah, there are this is memorized text. This is not memorized text in their definition of memorized text, but it is right. So it sort of mixes up different training data points together. And this I think is a strong, it's very strong evidence for how these language models work in that they sort of take training data points and they just kind of mix them together and they can do this in a grammatically well-founded fashion. They can also change individual words of a sentence and so on. By the way, it doesn't mean that people are doing anything smarter. Like there are arguments, like the best arguments I hear are, you know, people are kind of doing the same thing. They're just kind of recount the training samples in their a bit of their own words. But yeah, this I found extremely, extremely interesting. And also, you know, what I found from GPT-3 with this Google example was that the problem of memorization may even be way worse than what they analyze in this paper right here, because they look for sort of direct, direct overlap in text, whereas they wouldn't catch strings that are sort of reformulated. Again, okay, so here they say, lastly, they say they can extend text and this thing here I find very interesting. So they say, if they if they put in this prompt 3.14159, GPT-2 will complete the first 25 digits of pi correctly. Interestingly, when they input pi is this, it gives the first 799 digits. And if they say E is this, and pi is this, then it gets the first 824 digits correctly. So they make the point here that the memorization problem could actually be much worse if you only knew what prefix to input. So this strengthens my case for the future job description of a prompt engineer, right? It seems to be that it's quite a sort of magical power to know what to input into these language models to make them output what you want them to output in this context, but also in the context where you actually want to do them want want them to do something useful. Right. And here, here is where they investigate this number k. So you might have noticed and this is a bit of the criticism of my paper up until this point. Yes, they have you know, they have the k equals one right here. And they sometimes say that it's only found in very few examples. But essentially, they just they they they investigate this memorization here, pretty much in absence of k of what they themselves define to be problematic, right? They say, well, it's problematic if it only appears in few training examples. But the the analysis here is done quite absent of k very often. And here is where they investigate this. So this is also pretty clever that the the experiments here are fairly clever. They find a they find a one piece one document, a paste bin document. So the paste bin document, where that is sort of a JSON document, and it has lots of links. And I found the document is a giant document, okay. And it's a giant JSON document with these entries. So there's this entry, there is color and then link and then here, the URL would go on, right. And it is the in fact, the the only document in the internet, at least these these authors claim that contains these URLs. But many of the URLs are repeated many times. In fact, here you can see that these are the continuations of the URLs, right? This one, even though it's contained in one document, it's actually repeated 359 times, and so on. So this is a playground. They say, okay, this document was in the training data of GPT two. Here, we know how often each of these strings appeared in the document. So they can directly make an experiment. How often does a string need to be present for the model to memorize it? They simply order by the number of total occurrences right here, as you can see, and they ask each of these models whether or not it has memorized the string. And they do this by inputting this. So this is the input. And they simply sample, if the model manages to output any of these URLs, they consider that to be memorized, if not, then not. If it doesn't memorize it, they have a second trick that if model can get half a point, if they input this first random sequence, I think they input six tokens of this random sequence. And if then the model completes, then they say, ah, it has memorized it, right? So you can see right here, it appears that the this large language model needs this needs a string, let's say 20 times or higher for it to memorize it. And you can also see the trend right here that if you go to the smaller models, they need a lot more in order to memorize them because they have less weights, they can't afford to memorize stuff easily, right? They need to extract the pattern. So they'd rather forget about the string incur a loss and focus on other training examples. So yeah, two things in this direction, smaller models in this direction, larger models. So that means that something like GPT three will have this problem much more pronounced. So that's the bad news about this result. The good news about this result is that this is the case where you have fairly random sequences, right? These even you know, that if you tokenizing this is not going to be natural text, and there are these, you know, random, these Reddit URLs have these random prefixes. So this is very much this sort of outlier case. It's a pretty clever case study to find this document, I have to say, but it is sort of good news that this is not the usual case, this is really the case that this data is very, very prone to being memorized, right? Because it's not patternable. And it's very random. And yeah, so okay, so that was that was that. As I said, the amount of hedging right here is is really, really, like, it's a lot. They discuss what you can do with it, you can train with differential privacy, though that doesn't really help, as we said, because some of these strings are included in, you know, more than one time. You can curate the training data, which doesn't really help because the training data is too large. You can limit impact of memorization on downstream applications. So if you fine tune, but we don't know exactly what fine tuned models forget, and what they retain, or you can audit, which is essentially what this paper paper right here does. And that seems like that seems like seems like a good, you know, the best strategy we have so far is is to audit these models. And yeah, so I wanted to quickly check out also the appendix, the appendix here shows sort of these graphs for the other methods. And it is very cool. If you want to, you know, check that out. And it has sort of categorization of what they find as these memorized pieces of text. But what my main point was right here is that this paper shows a problem, let's say, with these large language models, namely that they memorize certain pieces of training data. While that sounds scary, I feel that the nature of the data that it remembers is very particular. So not you cannot extract any piece of training data, the nature is very particular. It's the sort of outlier ish training data points. And also, it very, very, very often isn't enough that it just is there one time. So even when they say this piece of information is only in one document, very often it appears many times in that document. That together with the sort of non pattern ability of the data that it memorizes right here, actually makes me fairly, fairly optimistic, more optimistic than I would have thought honestly about these language models. Yes, so we'll see what the future brings. As I said, this is going to be more pronounced in larger models. And this is not the only problem with these models, as my GPT three, Google search in that video shows. All right, I hope this was enjoyable. Let me know what you think and maybe check out the paper. Bye bye.
 Hi there. Today we're looking at extracting training data from large language models by
 what appears to be a big collaboration between corporations and academic institutions. There
 are almost as many affiliations here as their authors. So this is joint work between, you
 know, as you can see, many, many sort of institutions. And it is a pretty cool paper. So the high
 level topic is that these authors take large language models, as the title says right here,
 and train large language models specifically, and they're able to extract training data
 just from the trained model. In fact, just from the black box access to the trained model.
 And not only are they able to extract training data, they are able to extract pieces of training
 data, sort of verbatim, that have appeared only very few times in the training data. And
 they that's what they call a form of memorization. So they're able to extract these with a kind
 of pretty clever attack. So if you look at this prime example right here, they are able
 to query GPT two in this case, which is one of these large language models to output this
 piece of text. And the black stuff here is by the authors to protect the sort of privacy
 of this individual right here, this is though this is a real piece of text that they actually
 got out. And you can verify that. So they're able to extract this just from GPT two. And
 needless to say, this has consequences for security and privacy and so on. Because if
 you train one of these models with let's say internal or private data, user data, and so
 on, you have to be worried that these models are going to just output that data again,
 on the other end, and potentially leak information. This, of course, has not been a problem that
 much so far if you know, once we just trained image classifiers and so on. But here, especially
 with only black box access, this seems like it has some some consequences. So we'll go
 over the paper, we'll go over the the attack or the technique, the author's device, which
 is, I think, pretty clever. We'll go over sort of the results that they get from using
 this on a GPT two. And we'll go over my opinion of the paper, which I can already tell you,
 my ultimate opinion is that the attack is cool, the concerns are valid, but the paper
 is probably written a little bit more scary than it ultimately seems. In fact, I find
 the the results, the actual results of this paper fairly okay, like fairly promising,
 and sort of straightforward, not that scary. And also, the paper is interesting from another
 perspective, namely, from the perspective of what it tells us about these language models
 and how they work. And it it sort of strengthens a number of hypotheses that I've put forward
 in my video about GPT three, about how these models work. And that's also fairly cool to
 see in this paper. So we're going to jump in here. And as always, if you like content
 like this, don't hesitate to share it out, or subscribe and subscribe, I should say,
 if you're not yet. Alright, so they say it has become common to publish large, so billion
 parameter language models that have been trained on private data sets. This paper demonstrates
 that in such settings, an adversary can perform a training data extraction attack to recover
 individual training examples by querying the language model. Right, so we have a we already
 have quite a bit of information right here. So large language models have been, of course,
 trending with, you know, especially since GPT three, but at least since since the advent
 of the Transformers BERT and so on, though BERT isn't exactly a language model. So language
 models are models that, given a piece of text predict the next word, let's let's so easy
 as that or they predict the probability distribution over the next word. So if you say a cat sat
 on, so that's the input, the language model would give you a probability distribution
 over the next word. So the next word might be the or the next word might be a or the
 next word might be next, because of next two and so on. And it will sort of give you a
 probability distribution over each of these words that kind of looks like a face. It will
 tell you how likely each next word is and so on. And then you can sample from it, you
 can sort of choose one of those words and then go on. And you can evaluate the likelihood
 of entire sequences and so on. So GPT three is one of those large language models. And
 these large language models, they've been, of course, since they are large, we know that
 they also need a lot of data to be trained on. So a large language model would take like
 a giant piece, a database of training data, which is scraped from the internet usually.
 So this is too much to simply be curated by humans, they just let scrapers run over the
 internet, then they use this to train the model, whatever that is in GPT, GPT two in
 this case, and GPT two will then be a trained model. So you sort of throw the training data
 away, and you simply say, this is our model. Now, we're going to publish this, right. Now
 the problem is, if there is a piece of data in here, that is kind of secret. And you think,
 well, it's just one piece of data, like how much can how much can go wrong, right? The
 problem is, if I can inspect GPT two and recover this exact piece of training data, so that
 GPT two will output that exact piece, right, that is, is a problem. Now they make some
 good points here, this notion of a piece of training data, and what it means to memorize
 a piece of training data, and what it means to extract one is fairly fuzzy. And they go
 quite a bit deeper in this paper. So they have kind of strict definitions. They say,
 we demonstrate our attack on GPT two, a language model trained on scrapes scrapes of the public
 internet and are able to extract hundreds of verbatim text sequences from the models
 training data. These extracted examples include public personally identifiable informations,
 so names, phone numbers and email addresses, as you saw on the right here, IRC conversations,
 code 128 bit UUIDs, and so on. So they are able to extract all of these things from the
 trained model, right? And this, you can already see that how this can become a problem. They
 say our attack is possible, even though each of the above sequences are included in just
 one document in the training data. And this notion, this notion of memorization here,
 and when it is dangerous, they correctly say that this is only dangerous, of course, if
 the training example is contained in, let's say, only one piece of training data. Because
 if something is contained in thousands of pieces of training data, it's okay to memorize
 that, right? If a name of some famous person is memorized, and maybe the president of the
 USA lives at the White House, that it is not a secret, right? So it is okay if your language
 model remembers that, because it probably occurs in many training data points. However,
 if something is contained in just one document, right, and the model remembers it, then that
 is kind of true memorization. It is not maybe, or, you know, it's probably not learning anything
 from that data point, it's simply memorizing it to make its training loss lower. So that's
 the case on the right, right here. Though I have to say, this, as I said, it's written
 a bit more scary. So they don't exactly say that this name and phone number is contained
 in just one document. And they also say like, this is, of course, this is, this is on the
 public internet, GPT-2's training data was scraped from the public internet. So here
 is sort of my first investigation into this. First, you can Google this and you'll find
 it. You'll find this. And even though you know, the blacking out here also is a little
 bit of, I think it's a little bit gimmicky, because I don't see a problem with disclosing
 this particular piece of information. And I'll show you why. So when you search for
 it, you'll find the NIST homepage, you'll find a cryptographic algorithm validation
 program. And you'll find that this is a description of a software implementation. And here is
 the personally identifiable information, you can see, this is a corporate address. So this
 is a address of a corporation. And the contact information is a corporate contact is a corporate
 email address, it's a corporate phone number, and so on. This is the exact thing right here.
 And you know, with with respect to it only being present once in the training data. So
 if you actually search for if you complete the name here, and search for this, you'll
 find many, many, many, many, many results. Now, I don't know how many of these results
 are actually from, you know, in the GPT-2 training data, no one knows that, except OpenAI.
 So there's two Google pages of results. But oh, Google has D sort of D duplicated some
 of them. And now if I click on all, there are many there are 9000 results for this.
 And they are not all the same. Oh, no, no. So if you look at a bunch of those, you'll
 see that they are almost the same. But here, at the bottom, as you can see, this changes.
 So you know, depending on your scraper, these all count as separate websites. And therefore,
 I'm not so sure that this particular piece of information here is contained only once.
 Plus it is a corporate contact. So again, so to my point, the paper might be written
 a bit more scary than, than it ultimately turns out to be. Though, you know, you have
 to you have to make two different points like this particular piece of information. Yes,
 it might be written a bit more scary and gimmicky with the with the blacked out stuff. However,
 right? The paper has a point namely that if let's say you as a company do this on internal
 data, it might very well be. And they do have examples where they reproduce data from just
 one document. But even it might be that something like this happens to you internally, where
 you sort of maybe in your internal document base, you sort of do quasi duplicated document
 with the same information over and over. And and that's not the duplicated. And then your
 language model sort of memorizes that. So it's quite it, it has a point the paper. That's
 that's what I'm trying to say. I hope that's clear. Alright, so we'll get to the results
 in a bit. I hope I've already given you some sort of a taste for what you can expect. So
 first of all, they go into language models into sort of the definition of language models.
 And the language model here is simply framed as a model that can sort of give you a a probability
 of a sequence of text in sort of a stepwise fashion. So always probability of next word
 given the previous words, and you can evaluate that. Right, so the access to the model that
 they assume here is access to let's say the logits of the model or the output distribution
 of the model. And they say they use GPT two, because it's trained on large piece of text,
 but it's also you can you can evaluate it, it's not as slow, I guess as GPT three, and
 it's publicly available. However, the training data to GPT two is not publicly available.
 But they do have someone of open AI on the paper here. And this person at open AI made
 like mate, they could sort of query the open AI person to make sure a given piece of text
 that they find is or isn't in the training data of GPT two. So that's how they work.
 So that one per the open AI person acts as an API for the training data. Right, so they,
 they do, they define their attacks here. So they do a lot of things to, to set up cleanly
 what they do right here. So they have two points right here, there is this notion of
 memorization. Okay, so there's, they say there are many ways to define memorization in language
 modeling. In this particular piece of work, they say it is okay to memorize some stuff,
 they say language models must, for example, memorize the correct spelling of individual
 words, right, because the words are made of word pieces, and the language model needs
 to output that. So that's fine if it memorizes this. Indeed, there is an entire area of research
 that analyzes neural networks as repositories of memorized knowledge. For example, when
 GPT two is prompted to complete the sentence, my address is one main street San Francisco
 CA, it generates the next token 94107, a correct zip code for San Francisco in California.
 They say while this is clearly memorization in some abstract form, we aim to formalize
 our definition of memorization in order to restrict it to cases that we might consider
 unintended. So memorization as such isn't bad. What is bad is what they call here, the
 idetic memorization of text. So idetic memorization of text is when the model memorizes something
 that only appears very few times in the training data. So they say, we first define what it
 means for a model to X to have knowledge of a string. Our definition is loosely inspired.
 Yada yada yada, a model F knows a string, if s can be extracted by interacting with
 the model. So if you can input whatever you need to input, and the model outputs s, then
 the you say that model knows s, right. So if s is a piece of training data, then you
 say the model memorizes s, the model has memorized it. So here, they say a string is extractable
 from a language model if there is a prefix and the prefix here is the input to the model,
 such that if you input that model, the output will be the will be the string. And then they
 define this idetic memorization, respectively, they define k idetic memorization, a string
 s is k idetic, I have no clue whether I pronounce this correctly, k idetic memorized by a language
 model F, if F if s is extractable from F, so that's memorization, and s appears in at
 most k examples in the training data. Okay, so if this address of this person only appeared
 twice, but you could extract it verbatim from the language model, then that would be an
 example of two idetic memorization, okay, because k in that case would be two because
 it appears twice in the training data, though they they also, they are not clear what they
 mean by examples in the training data, because usually this training data is sort of chunked
 to make it fit into the language model and so on. And I think they do this on a document
 basis. So they would consider something like this here, one example, right, and then a
 different document, a different example. So if you have like, for example, if you have
 these IRC conversations that they are able to extract, so they claim here they are able
 to extract IRC conversations, or they're able to extract the user names of the IRC conversations,
 right? The user names might appear hundreds or thousands of time because they chat with
 each other. And it will all be, you know, in one document, but the document will be
 so long, they will actually be chunked into different training data pieces. Maybe I don't
 know. I don't know exactly what it means to be an example right here. But they do the
 example for sure, for sure, that piece of text can appear more than once, even if it
 is only in one example. In fact, they, they actually analyze the situation. Alright, so
 we've defined that this is the chi these k identity memorization. That's what we're looking
 for. That's sort of the problematic regime. If k is very small in the extreme k is one
 one piece of training data contains a string and we can extract the string at from the
 trained language model. They also say that for any given k memorizing longer strings
 is also intuitively more harmful than shorter ones. So this kind of makes sense. And they
 even they even go into sort of corner cases, they say that certain pathological corner
 cases, for example, many language model when prompting with the sequence, repeat the following
 sentence and then you give a sentence will do so correctly. This technically has any
 string to be known under our definition. But they, they of course don't do that. They assume
 they don't know the training data. So they can't just say repeat the following sentence
 and so on. But you do see that it is fairly hard actually to even define the problem right
 here, even though we as humans have a sort of an intuition what it means for a language
 model to unintentionally or un due to unintended memorization. Alright, so the adversaries
 objective here is to extract memorized training data from the model. The strength of the attack
 is measured by how private so how k idetic a particular example is stronger attacks extract
 more examples in total, and examples with lower values of k. They say we do not aim
 to extract targeted pieces of training data, but rather indiscriminately extract training
 data. While targeted attacks have the potential to be more adversarial harmful, our goal is
 to study the ability of language models to memorize data generally, not to create an
 attack that can be operationalized by real adversaries to target specific users. So you
 can see that here, they simply want some training data, they don't really care what it is, they
 simply want to get some so they're going to search for sort of the easiest to get training
 data. And that so they frame it as Yeah, we don't want to devise an attack that can attack
 individual users. But there is a different component to it. So if you had to sort of
 guess the password of any particular user, that would be you know, fairly, fairly hard.
 However, if you had to guess a password that was used by any user, it's fairly easy, right?
 Even if you discard the fact that most of people use password as password, and so on,
 if if people would just uniformly sample words from the dictionary as their password, still
 you'd have a decent chance of figuring out a password, right? We have a decent chance
 of figuring out, you know, not super high entropy things like maybe credit cards, you'd
 have a decent chance of figuring out the credit card number, just by guessing one. So this
 is the regime we are in here. And it's entirely different regime, I think, if you try to attack
 individual users. Essentially, what they're going to do right here is they're going to
 say, look, there's training data, right here. Now, some training data, these models can
 extract a pattern from, right? If and this is what we do with machine learning, right?
 We say, okay, this this data right here, they all have like some pattern. And this data
 right here is some pattern. And you can learn from this. And it has some pattern. So the
 machine learns to sort of abstract from extending data samples, and so on. But here is a data
 point that doesn't really fall into any of these categories. So what the model will do
 is it will simply say, well, this is its sort of own little group, I'll remember that I
 can extract some pattern from here and from here, but I can't extract any pattern from
 here. But I need to get my loss down. So I'll just remember that, you know, individual piece
 of training data. And that's exactly what we can recover with this sort of attacks these
 individual pieces that aren't really don't really have anything close, there is not really
 a pattern to it. So the best the model can do is remember that it doesn't mean that with
 this attack, you're going to get this piece of data or this piece of data, right. So if
 your personal identifiable information is sort of falls into some kind of regular pattern,
 it's, it's likely to be more safe against an attack like this. That's why they, for example,
 are able to extract these sort of UUIDs, or URLs with random strings in them, because
 random strings have no pattern, right. So they are likely to be out here away from the
 other training examples, where the best the model can do is actually remember the thing,
 rather than extract a pattern. Now, the other example here with this personally identifiable
 information, I believe that's just because it appears a lot of times, honestly, not because
 there is no pattern, but because it appears so many times that the model simply, you know,
 it's, it's, why should it extract a pattern when it appears so often, it can just, you
 know, remember it like a famous person's name seems to be an address that's important if
 it appears so often, I guess, from the point of view of the model. So that's, that's sort
 of what this does, again, it extracts indiscriminately, it doesn't mean that the attack can be leveraged
 to, you know, get any training data sample back. It's still worrisome, but you have to
 take into account. Another thing that that is really sticking out in this paper is the
 amount of hedging that this paper does. This, this almost in every paragraph, but certainly
 in every subsection, there is like hedging, hedging against, you know, why it is okay
 to publish this research, and so on. So, you know, when they say our attack target is,
 is GPT two, we select GPT two is a nearly perfect target from an ethical standpoint,
 the model and the data are public. So any memorized data we extract is already public,
 and so on. And they do this in in every piece of text. And, you know, in my video about
 broader impact statements, that was exactly my my point, these large corporations, right?
 If many, many of these authors, I think a fair amount of work went into framing this
 research, such that it sort of can't get attacked from, you know, people concerned about, you
 know, ethical considerations when releasing research like this, like this is clearly research
 that can be leveraged, you know, for for bad, if you will. But since these, you know, companies
 have a lot of resources, and, and there, you know, can put many people on this can devote
 fair bit of amount of of work into framing the problem that can be mitigated. Whereas
 if you know, some lonely PhD student would do the same research right here, the exact
 same research, I very doubtful it would be received as well as this piece right here.
 And in my opinion, as I already said in that video, this just sort of shifts, you know,
 a bit more power to these large institutions that sort of can afford the framing right
 here, they don't have to change anything about their research. But the rest of us do. All
 right, rant over. Let's continue. So they, they're going to do this in two different
 steps right here. And they have a diagram. Yes, I have a diagram. So first, they do this
 in two steps. Step one, they query the model, they have different queries, right, but they
 just sort of generate data from the model. So they generate lots of data right here,
 from the model. Then they select somehow they select from the model, a subset that they
 think these could be memorized training examples, then they do the duplicated, they they select
 again, and then they check, okay, this is it's fairly, fairly easy workflow. So step
 one is generate a bunch of data that you think could be memorized. And then step two, check
 whether you find these samples in the internet, because all of GPT two is training data comes
 from the internet. If you can find them on the internet verbatim, right, that probably
 means GPT two as remember, like the likelihood that it verbatim remembers, you know, I UUID
 that wasn't in its training data is almost zero. So yeah, this this goes by manual internet
 search. So respect to these authors who have done this, they start out with some fairly,
 fairly weak baseline, which is they simply generate the large quantity of data by unconditionally
 sampling. And then they predict which output contains memorized text by simply analyzing
 the likelihood. So whatever text the model finds highly likely, they they think that
 could be memorized. Because if you provide a model with training data, and you ask it
 to reduce its loss on the training data, it will assign highest likelihood to the training
 data. That's, you know, just, that's how these models work. So they assume that if a model
 has high likelihood, or low perplexity, that's the sort of same thing. Yeah, so you can see
 here, if the perplexity is low, then the model is not very surprised by the sequence and
 has assigned, on average, a high probability to each subsequent token in the sequence.
 And if that happens, they say, this could be memorized. This is obviously, obviously,
 very, very, very simple. See, this simple baseline extraction attack can find a wide
 variety of memorized content. For example, GPT-2 memorizes the entire text of the MIT
 public license, as well as the user guidelines of Vaughan life, an online streaming site.
 While this is memorization, it is only k-idetic memorization for a large value of k. These
 licenses occur thousands of times. The most interesting examples include the memorization
 of popular individuals, Twitter handles or email addresses. In fact, all memorized content
 we identify in this baseline setting is likely to have appeared in the training data set
 many times. So here they say it doesn't really work if you just sample and then look at what's
 most likely because yes, this will be memorized, but it is sort of a non problematic form of
 memorization like famous people's Twitter handles. That's this is like famous people's
 names at this point, right? So now they go about improving it. Okay, so they improve
 both steps, they improve step one. Where are we? No, it's down here. They improve step
 one by doing one of two things. Either you want your temperature to decay. So in this
 sampling, when you sample from the model, you have a temperature that you sample from,
 and you can decrease that over time. So at the beginning, you can let the model explore
 a bit, but then you can you can decrease it. And that's so the sorry, the the goal of changing
 step one is to create a more diverse set of generations, right? So you can sample with
 high temperature at the beginning, and then decrease it over time. Okay, such such that
 you still get sort of high likelihood sequences, but you get different ones. So you start off
 differently, and then you go into the high likelihood regime. The second way they change
 this is what they do is they go to the internet again. So they go to the World Wide Web, which
 is okay. I'm terrible at drawing the globe with okay, they go to the World Wide Web.
 And they just get pieces of text from the internet. So they get a website. And they
 just take some tiny substring from here, from this, and they use that as the input to their
 model. And that's sort of to get more diverse predictions. So if you input a short prefix
 that you found somewhere on the internet, and then let the model continue, that generates
 you have wide diverse variety of pieces of text. Okay. So that's how they up the DA how
 many different samples the model generates. Because in the initial experiments, they found
 that the model will sort of output the same things over and over again, if you simply
 query it unconditionally. So either high temperature or conditioned on internet text. The second
 step is sort of what I find the clever step. So here, they have to before they simply said,
 whatever has high likelihood, that's what we think is memorized. But of course, a lot
 of these will not be, you know, with low K memorized, a lot of them will simply be high
 likelihood because they're actually likely. So they say, okay, when, when is when are
 we in this situation? So let's say here is the here is our data set, okay. And here is
 the the MIT public licenses here. And it you know, it appears like billion, billion, billion
 times. So this data point is like ginormous. It's all you know, the MIT public license.
 And here is our outlier data point. Now, this model will extract patterns, let's say from
 this, and this is a pattern. And it will assign a single pattern to the MIT public license,
 because it just appears so often. And it will assign a single pattern to this data point
 down here, just because it's such an outlier, right? So how do we how do we devise a scheme
 that will find this one reliably, but sort of will recognize, wait a minute, this this
 memorization here is okay. But we need to devise a scheme without having access to the
 training data, right? If a human looks at it, of course, the MIT public licenses, oh,
 seems common, we know that it's common, and so on, we know that it's highly likely text,
 because it's a, it's a license almost everywhere. If a human looks at this right here and sees,
 you know, the name and address of a person or a credit card number, we know that's not
 really highly likely text. And that's sort of the answer right here. So we say if a human
 looks at it, but what is a human, a human is just another language model, among other
 things, right, but the human is just sort of another thing that has an intuition of
 how how likely text is. So the basis of their approach is going to be the following. Let's
 take a second, second data set, okay, sampled in the same way also from the internet, but
 not in exactly the same way. In fact, they use common crawl instead of the the Reddit
 outbound links that GPT two used. But we take any other data set, and I'm going to draw
 the other data set. So here's the data point, here's the data point, maybe this data point
 is duplicated from the other data set. And here's the data point here one, right, so
 you're going to have sort of other data points. But also, you know, since you're sampling
 from the internet broadly, you're going to have the MIT public license many times. And
 you're also going to have the outliers in this data set. Now, the important part is,
 you're probably if you sample this differently, in the same fashion, but a bit differently,
 you're probably not going to have this same outlier right here, you're probably not going
 to have that in your new data set. Okay, so you can see in the new data set, I hope you
 can see this, you're going to have the the same pattern extracted here, even though it's
 from you know, slightly different data points, you're going to have maybe a pattern extracted
 here, maybe one here, you're going to have this same cluster here, because the MIT public
 license will appear, even though it comes from other documents, it's copied over and
 you're going to have this outlier right here. So what you can do to differentiate our two
 our two things, you can consider a second language model. And you can ask, so here you
 have two things that the first language model things are very likely, you have this thing
 right here. And you have this thing right here, both the first language model consider
 super likely, you ask the second language model, and the second language model says,
 yes, the MIT public license, I consider that to be also super likely. But this outlier
 over here now that's I've never seen that what's that that seems very unlikely. And
 so by the ratio of the two likelihoods of the two different models, you can find out
 samples that the first model finds super likely, but the second model things are not likely
 at all. And that's exactly the trick they use right here. In fact, they use many instances
 of that trick. So here are the strategies perplexity is simply what they use before
 whatever is likely is probably memorized. This yes, it's memorized, but it's often
 memorized justifiably. Then they have these strategies small and medium. And and this
 is the ratio of the log perplexities of the largest GPT two model, that's the one they
 attack and the small GPT two model. And this ties into so you don't even need a different
 model, right, you can simply train a the reason they train a smaller model is the following.
 And we on the machine learning street talk podcast, if you don't know that it's a it's
 a podcast where we talk to people from various, you know, from the industry in from various
 research labs, and so on. And we spoke with Sarah Hooker, who we talked about their paper,
 the hardware lottery, but she also has other research, where she sort of shows that if
 you have weights, so you have a neural network, and it has, you know, layers, layers, layers,
 and you have weights in these layers, right? What she was able to show is that not all
 weights are equal. So some of the weights, let's say the weights here will be allocated
 to these pattern extraction things. So you know, here we have these, you know, you have
 data training data training data outlier outlier, right? So you'll have this, you have these
 weights representing this pattern within a layer, right? You have these, this pattern
 will be represented by these weights right here. And then you'll have other weights,
 they're sort of allocated to remembering single or very few outliers. Okay, so here, this
 will be allocated. And these will be disproportionate. So there will be many, many more data samples
 covered by, let's say, this piece of weights right here, I should have drawn the bottom
 one smaller than by this. So there might be, you know, 1000 training examples, covered
 by one piece of weight space. And there might be only one piece of training data covered
 by this other piece of weight space. And that's simply because it can extract a pattern from
 one but not from the other. So it needs to memorize it. And the larger we make these
 models, you know, the more parameters we give them, the more the more the more ability they
 have, the more space they have to do this remembering. So what what Sarah Hooker noticed
 in her paper is if you then distill these models, and distillation is the process of
 taking these models, and putting their knowledge into smaller models, then what happens is
 not all training data points will will so that in distillation, you usually lose performance,
 not all training data points will lose performance equally, namely, you will lose performance
 on the training data points that are sort of these outliers that are these not often
 represented in the training data that you know, the model has a harder time extracting
 patterns from it. So they will be seldom patterns, or just hard patterns, I would also assume
 that, you know, patterns that are harder to extract will also fall, fall away. So the
 the more complicated patterns will also be sacrificed. But I guess, among the things
 are these outliers. So if you train a smaller model, the smaller model would have less ability
 to remember these outliers. And therefore, if you do this, you don't even have to do
 it on a different training data set, right? You can simply compare to the same model trained
 on a sorry to a smaller version of the same model trained on the same training data set,
 because that will probably not remember the outliers as much. It would have been interesting
 if these authors here had actually distilled GPT to and though they do not have access
 to the original training data, so I can get why they didn't do it. But would be interesting
 to see that. That gives me an idea sort of maybe there is actually a way to look at the
 weights and I get these these authors don't have access to the weights, but maybe there's
 a way to look at the weights and to actually be able to sort of in some way spot right
 which of the which of the weights only are associated with with single or very few training
 data points. Maybe during training, you can sort of count how many times a weight is updated
 in a substantial amount or maybe looking at the attention matrices, you can sort of determine
 what are the kind of patterns that need to happen that lead to this weight being activated,
 right? So if there is a weight, and it's activated by lots of lots of different patterns, maybe,
 you know, that weight is useful for many, many forward propagated signals. But if there
 is another weight that's only activated by a specific pattern, right, then maybe that's
 one of these these memorization weights. So maybe there's a way to recognize these in
 the weights directly. So distillation appears to be sort of a defense against this this
 memorization of things, though that's not that's not done in this particular paper,
 they also have different strategies. So you don't need to do this neurally, right, you
 can compare the ratio of the perplexity that GPT two gives to the zlib entropy. So this
 is simply a text compression method, you can even compare it perplexities between the original
 string and the lowercase version, and so on. So they extract for each of these configurations,
 we select 100 examples among the top 1000 samples. So they produce 1000 samples, and
 they sample 100 from those 1000. So they mostly sample from low ranked samples, but also they
 explore some of the high ranked samples, they have a formula where they sample, they de
 duplicate, and then they investigate. All right, so they do Google searches, if they
 can find the thing, they say that's memorized. Alright, so they say, across all strategies,
 we identify 604 unique memorized training examples from among the 1800 candidates, our
 best variant has a true positive rate of 67%. That's quite remarkable, right? So 67% 67%
 of the things that this method delivers you automatically are actually memorized. Though
 you have to qualify that right? If you want more than 1000 examples, that rate is going
 to drop, right? You since you select the top 1000 examples, these are the most likely to
 be memorized. So yeah, if an attacker wants more, if they want to scale this attack up,
 their positive rate is gonna plummet fairly quickly, I'm going to assume it would actually
 be interesting also to see how that develops with the top the top retrieve document right
 here. But I get the they have to do Google searches to figure out and then ask open AI
 to figure out if if it's really a memorized training example. They say their categories,
 we manually group the memorized samples into different categories. The results are shown
 in table one, most memorized content is fairly canonical text from news headlines, log files
 entry from forums or wikis or religious text. However, we also identify a significant amount
 of unique data containing 128 bits UU IDs correctly resolving URLs containing random
 strings and contact information of individual people. Okay, so as I said, these, this is
 this is fairly interesting, but also a bit expected, right? If I give you the start of
 a UUID, then there is no pattern to extract, except I guess the UUID structure, but there
 is no deeper pattern to exact. So all the model really can do is memorize the UUID,
 especially if there aren't too many UUIDs in the training data, or if this particular
 UUID is some sort of, as I said, it's this outlier type of situations, the same thing
 for, you know, URLs containing random strings, these are just not pattern extractable, therefore,
 easily, more easily remembered by the model than learned. So you can see right here, the
 breakdown, where they see how many of what they extract. And here contact info, 32 named
 individuals non in non news 46. That's a fair amount of things you can extract from GPT
 two, you have to say that that is all right, all of GPT two, you get approximately 100
 things that are kind of names or contact informations. So as I said, not too bad, specifically considering
 what I've shown you here, right? They, that's one of these contact informations. And they
 do say this in the paper that this person, this information was obviously released in
 the context of this software project. And the problem is only the model might actually
 output this in a different context, right? The model might think, oh, now I need to output
 some sort of name and address, what kind of names and addresses to know what this name
 and address appears pretty often, I'm going to put that here. And so that's a failure
 case, you know, that these things can do. So here is a sort of a graph. And they have
 more of these graphs later. But you can see that here, for example, is a GPT two perplexity.
 And here is this z lib entropy. And if you plot them one against another, most things
 will fall on this diagonal right here with, you know, the giant blob around here for most
 texts of the internet. And there will be a region where GPT two things, this is fairly
 low perplexity. But z lib thinks the text is relatively high entropy. So these are candidates
 for memorization. And the red and blue here are the ones the authors selected for checking.
 And the ones that are blue are ones that they found are memorized from the internet. So
 a fairly high percentage, in fact, 67% of this method that they selected was, in fact,
 was memorized. Though, as I said, you can see that there aren't super many more, right?
 So this is this is all samples. And I don't know how many, you know, they could generate
 more, but you can see that it gets pretty sparse out here. Okay. Yeah, so examples of
 memorized content, personally identifiable information. They say there are several examples
 of individual people's names, phone numbers, addresses, and social media accounts. Some
 of this is memorized content is just exclusive to a few documents. For example, we extract
 the usernames of six users participating in an IRC conversation that happened in exactly
 one document. Yeah. So I guess the question is, how often did the usernames appear in
 that one document, right? And once the model sort of, and how, how distinct are these usernames
 from other usernames? Because if they're very distinct, and they happen, you know, they
 have a long conversation, it can be easy to see that the model will remember that not
 saying this is not a problem. I am telling you, the models, it's not, it's not that they'll
 just randomly remember stuff, there needs to be very specific conditions for the models
 to remember stuff. So they say, we identify 50 examples of memorized URLs that correctly
 resolve to live web pages. Okay, many of these URLs contain uncommon pieces of text, such
 as random numbers or base 64 encoded strings. Again, this this random element right here
 makes it you can't extract a pattern. They say we identify 31 generated samples that
 contain snippets of memorized source code. And they can actually extend that. So they
 can take these snippets and they always, I think they do 256 token length, but they can
 extend that to sort of verbatim recover the source code. And that's also you know, that's
 that's fairly interesting. And unnatural text, yeah, these UUIDs. A Google search for this
 string identifies just three document containing this UUID. And it is contained in just one
 GPT-2 training document. Okay, though, again, we are not seeing how often they say table
 three gives nine examples of k equals one memorized content, each of which is a random
 sequence between 10 and 87 characters long. You can see the table right here. So these
 are examples of random strings that for some reason appear in this training data in exactly
 one document. However, this string right here, for example, appears 10 times. And this string
 right here appears 311 times. So again, it's a random string that appears 10 times is fairly
 often for a piece of text to appear, especially the same piece of text that is not pattern
 close to any other piece of text. It seems okay that the model remembers that it seems
 expected, right? So yeah, here, they also say data from two sources, we find that samples
 that contain two or more snippets of memorized texts that are unrelated to one another. In
 one example, GPT-2 generates a news article about the real murder of a woman in 2013,
 but then attributes the murder to one of the victims of a nightclub shooting in Orlando
 in 2016. And this I found very, very interesting, right? Because that's exactly what I said
 GPT-3 does, right? Especially so in GPT-3, they have this example of GPT-3 writing an
 entire news article about, I'm not even sure about some pastors, some split in the Mormon
 church or something like this, or I don't remember correctly, but I was able to Google that. And
 I did not find the verbatim sequence, but I found that article that GPT-3 wrote many,
 many times in sort of different words in written down in, you know, books and reported about
 and so on. So what GPT-3 did is simply, I would guess interpolated between these things.
 And here they find the same thing GPT-2 just takes two pieces of text and sort of finds
 that they're close and sort of interpolates between the two. I would call this memorization
 two and they say, yeah, there are this is memorized text. This is not memorized text
 in their definition of memorized text, but it is right. So it sort of mixes up different
 training data points together. And this I think is a strong, it's very strong evidence
 for how these language models work in that they sort of take training data points and
 they just kind of mix them together and they can do this in a grammatically well-founded
 fashion. They can also change individual words of a sentence and so on. By the way, it doesn't
 mean that people are doing anything smarter. Like there are arguments, like the best arguments
 I hear are, you know, people are kind of doing the same thing. They're just kind of recount
 the training samples in their a bit of their own words. But yeah, this I found extremely,
 extremely interesting. And also, you know, what I found from GPT-3 with this Google example
 was that the problem of memorization may even be way worse than what they analyze in this
 paper right here, because they look for sort of direct, direct overlap in text, whereas
 they wouldn't catch strings that are sort of reformulated. Again, okay, so here they
 say, lastly, they say they can extend text and this thing here I find very interesting.
 So they say, if they if they put in this prompt 3.14159, GPT-2 will complete the first 25
 digits of pi correctly. Interestingly, when they input pi is this, it gives the first
 799 digits. And if they say E is this, and pi is this, then it gets the first 824 digits
 correctly. So they make the point here that the memorization problem could actually be
 much worse if you only knew what prefix to input. So this strengthens my case for the
 future job description of a prompt engineer, right? It seems to be that it's quite a sort
 of magical power to know what to input into these language models to make them output
 what you want them to output in this context, but also in the context where you actually
 want to do them want want them to do something useful. Right. And here, here is where they
 investigate this number k. So you might have noticed and this is a bit of the criticism
 of my paper up until this point. Yes, they have you know, they have the k equals one
 right here. And they sometimes say that it's only found in very few examples. But essentially,
 they just they they they investigate this memorization here, pretty much in absence
 of k of what they themselves define to be problematic, right? They say, well, it's problematic
 if it only appears in few training examples. But the the analysis here is done quite absent
 of k very often. And here is where they investigate this. So this is also pretty clever that the
 the experiments here are fairly clever. They find a they find a one piece one document,
 a paste bin document. So the paste bin document, where that is sort of a JSON document, and
 it has lots of links. And I found the document is a giant document, okay. And it's a giant
 JSON document with these entries. So there's this entry, there is color and then link and
 then here, the URL would go on, right. And it is the in fact, the the only document in
 the internet, at least these these authors claim that contains these URLs. But many of
 the URLs are repeated many times. In fact, here you can see that these are the continuations
 of the URLs, right? This one, even though it's contained in one document, it's actually
 repeated 359 times, and so on. So this is a playground. They say, okay, this document
 was in the training data of GPT two. Here, we know how often each of these strings appeared
 in the document. So they can directly make an experiment. How often does a string need
 to be present for the model to memorize it? They simply order by the number of total occurrences
 right here, as you can see, and they ask each of these models whether or not it has memorized
 the string. And they do this by inputting this. So this is the input. And they simply
 sample, if the model manages to output any of these URLs, they consider that to be memorized,
 if not, then not. If it doesn't memorize it, they have a second trick that if model can
 get half a point, if they input this first random sequence, I think they input six tokens
 of this random sequence. And if then the model completes, then they say, ah, it has memorized
 it, right? So you can see right here, it appears that the this large language model needs this
 needs a string, let's say 20 times or higher for it to memorize it. And you can also see
 the trend right here that if you go to the smaller models, they need a lot more in order
 to memorize them because they have less weights, they can't afford to memorize stuff easily,
 right? They need to extract the pattern. So they'd rather forget about the string incur
 a loss and focus on other training examples. So yeah, two things in this direction, smaller
 models in this direction, larger models. So that means that something like GPT three will
 have this problem much more pronounced. So that's the bad news about this result. The
 good news about this result is that this is the case where you have fairly random sequences,
 right? These even you know, that if you tokenizing this is not going to be natural text, and
 there are these, you know, random, these Reddit URLs have these random prefixes. So this is
 very much this sort of outlier case. It's a pretty clever case study to find this document,
 I have to say, but it is sort of good news that this is not the usual case, this is really
 the case that this data is very, very prone to being memorized, right? Because it's not
 patternable. And it's very random. And yeah, so okay, so that was that was that. As I said,
 the amount of hedging right here is is really, really, like, it's a lot. They discuss what
 you can do with it, you can train with differential privacy, though that doesn't really help,
 as we said, because some of these strings are included in, you know, more than one time.
 You can curate the training data, which doesn't really help because the training data is too
 large. You can limit impact of memorization on downstream applications. So if you fine
 tune, but we don't know exactly what fine tuned models forget, and what they retain,
 or you can audit, which is essentially what this paper paper right here does. And that
 seems like that seems like seems like a good, you know, the best strategy we have so far
 is is to audit these models. And yeah, so I wanted to quickly check out also the appendix,
 the appendix here shows sort of these graphs for the other methods. And it is very cool.
 If you want to, you know, check that out. And it has sort of categorization of what
 they find as these memorized pieces of text. But what my main point was right here is that
 this paper shows a problem, let's say, with these large language models, namely that they
 memorize certain pieces of training data. While that sounds scary, I feel that the nature
 of the data that it remembers is very particular. So not you cannot extract any piece of training
 data, the nature is very particular. It's the sort of outlier ish training data points.
 And also, it very, very, very often isn't enough that it just is there one time. So
 even when they say this piece of information is only in one document, very often it appears
 many times in that document. That together with the sort of non pattern ability of the
 data that it memorizes right here, actually makes me fairly, fairly optimistic, more optimistic
 than I would have thought honestly about these language models. Yes, so we'll see what the
 future brings. As I said, this is going to be more pronounced in larger models. And this
 is not the only problem with these models, as my GPT three, Google search in that video
 shows. All right, I hope this was enjoyable. Let me know what you think and maybe check
 out the paper. Bye bye.