Author Interview - Typical Decoding for Natural Language Generation
#deeplearning #nlp #sampling

This is an interview with first author Clara Meister.
Paper review video hereé https://youtu.be/_EDr3ryrT_Y

Modern language models like T5 or GPT-3 achieve remarkably low perplexities on both training and validation data, yet when sampling from their output distributions, the generated text often seems dull and uninteresting. Various workarounds have been proposed, such as top-k sampling and nucleus sampling, but while these manage to somewhat improve the generated samples, they are hacky and unfounded. This paper introduces typical sampling, a new decoding method that is principled, effective, and can be implemented efficiently. Typical sampling turns away from sampling purely based on likelihood and explicitly finds a trade-off between generating high-probability samples and generating high-information samples. The paper connects typical sampling to psycholinguistic theories on human speech generation, and shows experimentally that typical sampling achieves much more diverse and interesting results than any of the current methods.

Sponsor: Introduction to Graph Neural Networks Course
https://www.graphneuralnets.com/p/introduction-to-gnns?coupon_code=SUNGLASSES&affcode=999036_lzknae-d

OUTLINE:
0:00 - Intro
0:35 - Sponsor: Introduction to GNNs Course (link in description)
1:30 - Why does sampling matter?
5:40 - What is a "typical" message?
8:35 - How do humans communicate?
10:25 - Why don't we just sample from the model's distribution?
15:30 - What happens if we condition on the information to transmit?
17:35 - Does typical sampling really represent human outputs?
20:55 - What do the plots mean?
31:00 - Diving into the experimental results
39:15 - Are our training objectives wrong?
41:30 - Comparing typical sampling to top-k and nucleus sampling
44:50 - Explaining arbitrary engineering choices
47:20 - How can people get started with this?

Paper: https://arxiv.org/abs/2202.00666
Code: https://github.com/cimeister/typical-sampling/blob/3e676cfd88fa2e6a24f2bdc6f9f07fddb87827c2/src/transformers/generation_logits_process.py#L242-L272

Abstract:
Despite achieving incredibly low perplexities on myriad natural language corpora, today's language models still often underperform when used to generate text. This dichotomy has puzzled the language generation community for the last few years. In this work, we posit that the abstraction of natural language as a communication channel (à la Shannon, 1948) can provide new insights into the behaviors of probabilistic language generators, e.g., why high-probability texts can be dull or repetitive. Humans use language as a means of communicating information, and do so in a simultaneously efficient and error-minimizing manner; they choose each word in a string with this (perhaps subconscious) goal in mind. We propose that generation from probabilistic models should mimic this behavior. Rather than always choosing words from the high-probability region of the distribution--which have a low Shannon information content--we sample from the set of words with information content close to the conditional entropy of our model, i.e., close to the expected information content. This decision criterion can be realized through a simple and efficient implementation, which we call typical sampling. Automatic and human evaluations show that, in comparison to nucleus and top-k sampling, typical sampling offers competitive performance in terms of quality while consistently reducing the number of degenerate repetitions.

Authors: Clara Meister, Tiago Pimentel, Gian Wiher, Ryan Cotterell

Links:
Merch: http://store.ykilcher.com
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi, y'all. This is an interview with Clara Meister, who is the first author of the paper Typical Decoding for Natural Language Generation. This paper, I believe, is really important because it presents a new sampling method that makes language models output much more human-like texts. I've already made a review about the paper if you haven't seen that yet. Check it out. Clara has seen it and we're able to dive directly into the matter. This interview was very cool. I learned a lot. As always, if you like, leave a like, tell me what you think in the comments and I'll see you around. Bye bye. Hey there, today's sponsor is the course on Introduction to Graph Neural Networks. This is a course by my friend Zach Jost, who is an expert in graph neural networks. He's packed all his knowledge into one course that will educate you on both the theoretical and hands-on practical aspect on graph neural networks. Graph neural networks are really important. They're definitely one of the most interesting areas in deep learning right now. They've also powered a lot of recent advances in scientific breakthroughs, such as alpha fold protein structure predictions or better traffic predictions. If you use my link, you'll get a 15% discount on the course. Enrollment is open right now and lasts until April 1st or until spaces run out. All right, let's get into the video now. See you. Hello everyone. Today I'm here with Clara Meister, who is the first author of the paper, Typical Decoding for Natural Language Generation. Clara, welcome very much to the channel. Thank you. And thank you for having me. This was a really neat paper. I have to say I have just finished my last interview, not just now, but I finished my last interview about a system called BLEP. What they said is essentially they have a system that generates captions for images in an automated fashion. Then they have a filter that kind of weeds out the crappy captions. They use that as a means of generating more high quality data. They and many others before them have found that how you sample from a model, like from the language model they've trained, matters a lot. Specifically, they told me that nucleus sampling in their case was really a defining factor in getting more of a diverse sample set. They particularly compared it to greedy sampling and to beam search, which they found super underwhelming. I've come across a lot of systems in recent times, for example, AlphaCode as well. I don't know if you know how exactly AlphaCode does what it does. I don't either, but from the paper I could gather that they sample a lot of potential solutions and then they reduce those down by filtering and clustering. Again, they rely heavily on being able to sample diversely and to sample many, many different things. I've for a while now thought maybe our sampling objectives are wrong for certain applications, namely for the applications where we actually are interested in more of a diverse output rather than the most likely output. Along came your paper, which essentially exactly plays into this and suggests a new method. I was super happy to see this. I think it really hits a nerve of the time. If you would pitch it, like the elevator pitch for the paper, what would you say about it? Yeah, I would say that specifically for language generation, I think with these large models that we've been training, that when we're generating language from them, we need to take into account what we really want from the model, what our objective is. Also, what we just normally do when we're speaking, when we're writing, how we use language. Trying to think about having this, what these models are is essentially probability distributions over strings. That's kind of a strange concept. It's not probably how we imagine language in our heads. There is some evidence in psycholinguistics that that's kind of actually a pretty good metaphor for how language is represented in our head. How we then go from that to generating language and what the characteristics of the language that we typically generate are, I think we really want to take that into account when we're trying to generate language from these models. If you just ask me to say something randomly, what am I going to say? I'm probably going to say, I don't know. I don't really have these really common phrases. But if we want something more interesting, if you want me to say something more interesting, then I'm going to not just pull the most likely sentence out of thin air. I'm going to try to convey information in what I'm saying. I think that these models have sort of learned how to do that implicitly. We can ask them then to try and do this in a similar manner to how humans do. Yeah. So you pretty quickly get to this notion of typicality, which is a notion from information theory. You connect it to various disciplines in psycholinguistics. But a typical message as far as I can understand it is, well, as the name says, one that you would expect to see from sort of a communication apparatus. But it is, do I understand this correctly, is one that you expect to see if you assume that the communicators want to transmit the optimal amount of information? Is this the core assumption behind how we think about communication between humans? Yeah. One important thing is typicality in the context of communication channels is really only defined in the context of a message here, some sort of message that you're conditioning on and trying to convey. So in here, I mean, especially when you're sampling from a language model without having this implicit message that you're conditioning on in the background, I think it's kind of hard to really quantify what a typical message in natural language should be. And I think we're very careful to say that there is this nice intuitive link between typicality and how humans use language and what type of strings we might expect when using natural language. But there's a lot of aspects of human language that don't really fall into the paradigm that you can really apply typicality to. And so you inspire, let's say, by this notion of typicality, or you're inspired by. So you define the notion of a typical message, and that is sort of the average information content you would see. I made a bit of a characterization in my video. By the way, we have to inform the viewers that I use the old archive version, and you just updated it. And you corrected essentially all the little criticisms I had about notation and things like this, just to get the lore right. It wasn't me that caused it. You did it ahead. And then I used the old version. You know, props to you for picking them out. My advisor always says that every single paper out there pretty much has math errors in it. Oh, yeah. Don't worry. It takes a critical eye to find them. It's super easy to just glance over them, not realize them. Well, I think it was actually straightforward. The paper is really easily readable. So when we think about how humans communicate, and let's assume for a moment what you say that in your hypothesis here, any given word should have an information content close to the expected information content, i.e. the conditional entropy given prior context. In other words, we expect this difference to be small in human-like text. And you also say that the human goal over here is to transmit information effectively while also minimizing the risk of miscommunication. I made a bit of an example right here as if I explain math, or if I explain the chain rule to someone who does and does not understand math, is this an appropriate example? Is this an appropriate metaphor for what you're going for? Or is this totally off? No, I think in a way that's right. I think that's actually perhaps even more related to what we described later on, which is the rational speech act, which is how we also are taking into account the listener when we're forming our messages. That's definitely a component that's taken into account. So we'll modulate the amount of information that we are conveying to basically account for what the other person might know. And I think that you can kind of model that in different ways. You can say that, in your case, I think how you put it, I think is a totally valid way to see it. In that case, we can say that the information content for the speaker is going to be much higher than for someone else. So I mean, yeah, I think that's a good comparison. So this notion of the expected information content is pretty important here. And we say, okay, if I'm at a certain, let's say I've uttered half a sentence, and then I look at the distribution of the next word. And that distribution is just the distribution of the language itself, if I understand this correctly. So I have my training corpus, which supposedly is all of human language, I analyze it in my head, I determine what's the conditional probability for the next word in the training corpus. And then your claim is that what I do is I don't actually sample from that distribution, I'm going to adjust in inside of my head, the distribution that I sample from two, two words that closely match the expected information content. My question is, why, why do I do that? Like I see the problem with always picking the highest likely word, right? If I if I have a broad distribution like this, I don't want to do that. I don't want to just pick the most likely one. However, why can't I just sample from this distribution? It seems like enough times I would actually, you know, pick some other words that is also completely fine. Yeah, I mean, so first of all, I think one thing is, when we're forming language, we are, I mean, we arguably aren't like sampling from this distribution, right? We kind of know, I mean, maybe to some extent, we're sampling what we're going to say next. But I mean, I think the important thing to internalize is that we have a message that we want to convey right every time that we're using language. And the way that we choose to do that is like at a specific information rate, because we want to communicate efficiently. But we also want to make sure that our message gets across without like having to repeat ourselves or confuse someone or, you know, making them like spend an inordinate amount of time processing what we're saying. And so because of that, like we're not going to choose super low information words all the time, because that's just kind of inefficient. Yeah, like, I can I can say all these filler words, right with and still get across a message, but adding like, it's like that, you know, that person that takes forever to explain something just goes about it in a super, like slow and redundant way. Don't make fun of my videos. What are you talking about? So I think that's something to to think about. And then sorry, the second part of your question, I've already forgotten. I mean, I so I think I've what I've understood is that if we look at just the distribution of the next word, that is, in all of language that is across humanity, everyone who's uttered ever that first half of the sentence, this is the distribution of next word. However, when I consider that I actually have a message to convey, that distribution changes, right? Is that about the characterization of what, like, my question would be, why don't I just sample from this distribution right here, given that if you know, many words are possible, it will actually result in kind of a diverse sampling. Yeah, I mean, I think that you like, first of all, I actually do think that in the case of like a perfect language model that you could actually sample from this distribution and be fine. I think that there are some there are some artifacts that are a bit strange, like especially in models that aren't trained as well with like this this long tail distribution that like that tail isn't necessarily learned all the learned very well, like what those actual probabilities are. And so, you know, you end up with like, just oddities. And, but beyond that, I mean, I do think that, like, we're not. I mean, we are trying to modulate when we speak, like the amount of information that we have per word, right? To keep it even. And this is this is not I mean, this is something that is perhaps not very obvious, but it is something that's like well studied in psycholinguistics, like how we how we convey a message. And like the coding that we will use within natural language. And so, like, yeah, we we we take this into consideration when choosing the next word. Yeah, not to be too redundant or to be too surprising. Yeah, and to end, again, to transmit what we actually want to transmit, right? Because I have something that I want to say, and that means I can't just blindly sample from the distribution, I would never actually transmit what I wanted to say, would it be would it be possible that, let's say, if I could hypothetically determine, you know, what what kind of let's say I have a message I want to transmit, could I somehow define the information content of the next word, given the message I want to transmit, and maybe also given the sentence, you know, so far t smaller than or smaller than t. Well, that's, I mean, that's actually usually what we're we're doing. And in so in a task like abstractive summarization, which, you know, we see is something that we experiment with, we are conditioning on that message, essentially, you know, a message being the article, right. And so it is like, we are taking that into account when we're trying to build our next word. Yeah, and it is still like, this distribution should reflect the fact that there is a message that we want to convey. And, you know, given that message, it sort of, it sort of reflects that, you know, maybe this word that without that knowledge would have been very surprising. But like, with that knowledge, with knowing that, like, we want to transmit this message, actually, that word is like what we would expect. Yeah. Okay. My my question, what I'm trying to get at is, if I train my language model for abstractive summarization, right, the conditioning of the message is maybe already in not maybe in here, if I use a decoder only model, but like, my question is still, why is this distribution here not enough? Like, why, why do I need to cut out the most likely things? Even though, you know, sometimes I actually want to say them. So, I mean, I think it's just to be more human like. Yeah, that's okay. That's the most I can say is, yeah, it's a it's fine, it's fine. So, you you make you come up with and we're gonna, we're gonna go back to these plots because I find them super interesting as well. You define this typical sampling strategy where you say, okay, we we have this thing here, which is the expected information content of the next word. And then we're just trying to as closely as possible match that. So we're just going to select a subset of all the words that we could pick, which closely match that expected information content according to your hypothesis. And then we're going to sample according to the new distribution that only consists of the subset of these words. So in the video, I think I raised a point which is maybe more of a, I don't know if it's circular logic or a philosophical point. But all our training data, presumably of these language models comes from humans, you know, using language transmitting information. Therefore, right? Shouldn't like if I now train my language model, and I use your method to sample things, and you claim it's a human like way of sampling things, shouldn't that a result in the same distribution? And B, shouldn't it sort of the expected information content if I measure before and after, like if I measure it in the training corpus, and then if I measure it as an output of my model, shouldn't that be the same? Because presumably the training corpus is already generated from humans. I mean, yeah, I think like, yes, I think that makes sense if I'm understanding correctly. And I also think we're kind of seeing that like in the earlier plots, we're actually seeing that, like, if there is like an average amount of information, right, according to the model, there's an average amount of information that each word will contain. And I mean, human text seems to be coming from quite close to what the model has learned that average information rate to be. And do you, did you investigate the outputs of your model? And sorry, sort of redid those plots on the output of your model and observe the same, the same pattern? Yeah, so that's like, yeah, that's something we did as well. We looked at basically a few different decoding schemes and saw what the, what these distributions looked like for the outputs of those decoding schemes. And I mean, things like, you know, to nucleus sampling with like very popular, you know, popular values of P looked similar. And so did the ones from typical sampling. We didn't, I think, honestly, they do look, they by visual, like visually, they look pretty similar, which is nice. It's also nice to see that sort of these more, these vetted decoding processes that have like stood the test of time are also actually mimicking these distributions. I think that if we wanted to be robust about it, we'd probably want to, you know, come up with some sort of quantification for how different these distributions are. And use that perhaps to see if that correlates with how well these decoding methods perform in terms of things like human evaluations. So can you tell us the story behind these plots a little bit more? Because you define epsilon in terms of an absolute value yet here I see values that are less than zero to both sides. So I didn't know which one is which. What's epsilon here? I tried to make it clear in the caption of the text, but I don't think I did. I mean, if I guess correctly, it's the conditional, it's the expectation minus the actual information. No, so it's actual information minus... I would have gotten it wrong. Oh, wait. No, no, I think you're right. No, no. Maybe you can tell us what does it, because these are kind of, so it's more, if I see this correctly, more sort of mass on the left side of these close to this boundary, which is really interesting. And then there's a long tail on the right hand side. What does that tell us about human language? I mean, that's like a very deep question. And I'm not entirely sure about what the shape of this distribution means. And I think it's very interesting that this is the shape of the distribution. And actually, we used a few models here, and all of them kind of did look like this, where you had this peak and then sort of a long tail. And yeah, I mean, I think that that's an investigation in its own right about how humans use language. So yeah, by the way, it is information content minus entropy. So remember, so low information content, high probability, right? So actually, human language tends to be to the like on the higher probability side of conditional entropy. This thing right here. So if we if we're way out on the right, it means that we actually transmit a lot of information actually more than would be expected. So there is it doesn't that there is a long tail of very high information words, let's say, do you think so because you in one thing that I skipped over that in the video review, but you make this point of humans, what they probably do is they want to everywhere in the message, they want to have kind of a constant information rate. So every word should approximately transmit this this expected information. So as you go through the sentence, do you think this could be violated a little bit because humans, most of them do tend to have like a short term memory of three to four words or so that they, you know, can keep keep ready in the sentence, maybe I can transmit this super high information word. And then before my receiver gets super confused, I can follow that up with like two or three clarifications, which which would be then maybe here in the lower information content, but they would be more. Yeah, I mean, so, like, I think it's hard to always avoid moments of high information. I mean, for example, if you're giving if you think about this very literally, in terms of like what those words could be, you know, they could be like someone's name, right. And that's kind of like you're introducing someone that's always kind of going to be like a high information moment, right. You have to remember, I mean, we always forget people's name, people's names, obviously, there's like, there must be a lot of information in those names. So very off the cuff explanation. But I mean, yeah, so I think it is hard to just 100% of the time, avoid those instances. But I mean, this is talking about sort of on average, what we're doing when we're constructing language. And I mean, so I guess I couldn't say whether in those moments, we want we try to perhaps on either side, balance out, like with lower information words, this high information word, because I mean, you know, maybe maybe we do in order to give the listener some time to internalize this information. But there are also especially with with speaking, which is a different domain than writing, right, there are other ways that we can modulate high information words, right. So we can elongate our speech to basically spread out information over time, right. And so it's not like here, we're just evaluating text. So, you know, we, I think, especially in text, we're going to see these longer tails, because you can't sort of distribute information over too many words in certain cases, like in the case of introducing a name. Yeah, I think that's and also it has to be said that, you know, you can, if you go to the left, you get into the super low information words. And there is only that many of them, right? As soon as I'm at the and, uh, right there, there aren't that many. However, there is, in fact, a long tail just in the language of super high information words that are quite unlikely. So maybe that plays a role into it as well. About these plots, you say you draw two, two different conclusions right here, which the first one is the peak nature reveals that humans indeed tend to form language with per word information content quite close to their expected information content. So this is kind of, you know, here is data that shows our hypothesis is correct. And the second one is the centering of these distributions around a value close to zero reveals that our probabilistic language generators are learning what this rate is, which, and my point was a bit when in order to make point one, you need point two as an assumption, right? You need, you need to claim, well, I can only say this because I assume our language models are modeling the probabilities of language well enough. Otherwise I could not conclude point one. Likewise, you couldn't conclude point two without having point one as an assumption. Is this, am I overlooking something here? Well, so, I mean, I think the point here that we wanted to get across was really that, you know, two things should be looked at in these graphs, which is the centering of the graph and also the shape of the graph. And I mean, so I think there is, there is an assumption that kind of has to be made here. I don't think it's as quite as severe as, as what you've mentioned, but I mean, it is sort of that this enter, this information rate is kind of a ground truth of sorts. But I mean, you know, you could, for example, shift, like you could shift to that entropy rate. You could shift the entire distribution and still, you could shift H and all the P's and you know, all of, all those numbers and still technically get the same distribution. So that I agree with. But like, I mean, I think like looking at the peakiness of it, clearly we're seeing that, you know, humans are generating language around a certain... Something, right?...content. Yeah. Yeah. What if it were centered around two instead of zero, right? It would be as peaky. Well, yeah, I mean, yeah, as peaky then like, yeah, like we'd probably be, that'd probably show that humans communicate at like a very low information rate, right? Or, yeah. So, but no, I mean, it's around, like it does seem to be close to this expected information rate. And I think one other, like the part two is really trying to show that like there's this, we would expect that if our model understands that, you know, humans are speaking at around an average information rate, that this distribution would be centered around, like on average, it would be predicting that information rate for a given word or like that information content, that probability for a given word. And it does seem to be doing this. Cool. Yeah, this is just a bit of a nitpick for me. I'm totally on board with, I mean, it's pretty clear the language models do model these probabilities relatively correctly, especially the ones with the higher probabilities. And I'm fairly convinced by these plots that what you're doing is something sensible. Yeah, no, I mean, I think you bring up a really important point. And I actually, like I'd spent a long time thinking about whether or not it was too circular, like, you know, whether you could have one without the other, really. And I mean, I think, like, I think at some point I came up with some examples, like some counterfactual examples where actually you could have one without the other. And of course, now, like, I can't remember what they are. But yeah, it's, it's, it's, I think, I think people understand what you're, what you're saying. There's definitely like a degree of freedom there, right? There's definitely something that could change that, you know, you could get those same results. And I think, but I think, like, that thing that could change would be whether the information rate learned by the model is like the quote, human information rate, the actual human information rate. And I'm actually not entirely sure that's important. It just has to be, it just has to get it right, like relative to what it's predicting the probabilities for words, right? Do you want to tell us a little bit about the experimental results? Because I have not gone into these at all during the paper review, things that you would like to highlight or anything like that? Yeah. So, like, as Yannick mentioned, there's a new version on archive, where we are, we also present a few different values for nucleus and top K, as in like the same, you know, same number of values. Oh, yeah, the hyperparameters. Sorry about that. No, no, I think it's very reasonable. I mean, the thing is, like, you know, there were only so many human evaluations we could afford. And we thought, like, you know, we should probably test out more values of our own method, since no one has done this before. But like, a lot of people have looked at nucleus and top K sampling. But then once it seemed like, okay, this is worth, this is research worth doing, we were able to get a little more money and launch a larger human evaluation. So those results are now in the paper. I mean, I think one thing that was really interesting for us was actually just the variety of values of tau that worked well. I mean, basically, like, most values of tau worked well. There wasn't like a huge difference between all of them, which we thought was really cool, because in comparison to nucleus and top K sampling, those methods were really dependent on N and K. And I mean, I think there was like a little, like, if you just look at the output of these models, you know, if you have a large tau, then maybe qualitatively, you could say that the text is like a little more normal, like a little more standard, and then maybe a little more diverse for low values of tau. But I mean, basically, it was just for, it was just interesting to see that for these two tasks, at least, that, you know, variety, like it wasn't, you didn't really need to tune tau that much, just kind of, kind of worked. It's important, right? Because that's one of the issues with these things is that if I have to tune the thing to every new task I do, I'm a lot less certain in, you know, kind of the generalization of this even within the same domain. But if it's interesting to hear and if it's really a kind of a handle on the craziness that I get out of these models, that could actually be even a cool property, right? If you say, actually, most values work, but it is, you know, it changes just the style. I think that that is a useful hyperparameter rather than a nuisance like in nuclear sampling. You know, if I don't get it right, it's going to be crap. Yeah, well, I would like to think that that's the case. I'm slightly biased here. Yeah, is there any, I mean, you run various automated tests in abstractive summarization and story generation. Most of the time, the typical sampling is on top of the pack, sometimes not, especially here in the story generation on some of these automated evaluations. Is that kind of an interplay between the evaluation, how the evaluation is done and the methods? Or if that is that a property of the task itself? What can you tell us about this? I mean, so I think a lot of these metrics, I think a lot of these metrics can only tell us so much. And, you know, the text that we end up generating, how it performs in terms of these metrics, I think like you'll see, for example, in human text, you'll get reasonably different values. Like you can get reasonably different values for things like repetitions within reason and the text be equally as good, at least qualitatively. So like, I think the important, I don't know if it's important is the correct word, but one of the critical things for us was like looking at whether we could avoid this really degenerate behavior with models. Because I think that's something that's like one of the bigger problems in language generation is just like this tendency for these methods to fall into repetitive loops. And I mean, we basically just like, we didn't really see any of that in using our method. And so I think that was an important takeaway. So yeah, I mean, always kind of performing well in terms of this, in these metrics that show how repetitive or redundant text is. I think it is what we would expect, right? You know, we're saying that like if text is, we want text to be about as redundant as human text is, because that's like one metric you can use to quantify information content, right? So it was good to see that that like, at least, it's a necessary, not sufficient criteria, but it was good to see that it was met. Yeah, I was just looking, like just now looking at perplexity, and yours is in bold. And I was like, wait a minute, lower perplexity is better usually. But then I realized what you have to do here is obviously match the perplexity of the reference text as closely as possible. So the goal is to be as close as possible to that number, which is really astonishing to see because in machine translation, people are fighting for 0.1 perplexity or so for the new state of the art. And here it's a difference of, it's quite a magnitude of difference between these methods, which is cool to see. And I think shows quite well that in something like story generation, these models might really just not, overfit is the wrong word, but overproduce not as creative outputs, or maybe even degenerate ones, as you say. I mean, I think actually in the context of machine translation, and this is something that an experiment that I want to personally perform is look at what the average perplexity of the reference text is, right? I mean, so and the generations, right? So the one thing about machine translation is typically we're evaluating on things like blue, right? Not perplexity so much that we're evaluating on the generations themselves, rather than the evaluation of the reference text, like what the perplexities are. But I mean, it would be, to me, it would be interesting to see what the perplexity of good generated text is compared to human like text. And I think in that case, they would actually probably both be quite small. At least that's my intuition. Of course, one artifact that I think would kind of get in the way of these experiments is the fact that machine translation often uses label smoothing, right? And label smoothing is basically like a form of entropy regularization. So it makes these distributions higher entropy even if they shouldn't be. And that actually, I mean, basically, you can read other papers about this that will explain it. But it is kind of it does interact with beam search. It's like the match of beam search plus label smoothing tends to work quite well. But I think if you were to really perform these types of experiments to understand what the types of perplexities for machine translate, like for translations, good translations would be, I think, yeah, you'd need to do it with a model that doesn't that hasn't had this sort of artificial inflation and entropy. Do you think our training objectives are the correct ones? Let's think of something like story generation is pretty, because what I'm hearing now is that, well, label smoothing but plus beam search works, but it's more like a hack to get around the weaknesses of beam search without label smoothing. Do you? And that is, you know, something I can maybe, you know, get get behind. Do you think we have the correct training objectives if our goal is really to create diverse and interesting set of outputs? Do you think it's a good strategy to train, let's say maximum likelihood, and then sample using something like typical sampling? Or should we also change our training strategy? So I personally think that maximum likelihood is a pretty robust objective. I mean, in terms of like the information theory perspective, I mean, when you when you are maximizing likelihood, right, you're also minimizing KL divergence. So you are basically looking for the model that assigns the same information contents to strings as as the empirical distribution. Right. So it's like they're just equivalent. And so I think if you take that into account, basically, if you take into account exactly what you're doing with your objective, and then from that, you know, go on to, okay, well, given given this distribution, right, how how would we go about how would like we as humans go about generating from this distribution? Or you know, how would if like you're generating an image, like how would nature go about like generating from this distribution? I think, you know, it's really important to I don't think there's a correct way necessarily to go about training and decoding. But I think we really need to take into account more their interaction and understand like, what is going on within that interaction. Yeah, I mean, I'm all on board, because it also means that we can use we can reuse the same model for multiple, let's say tasks, if we swap out our decoding strategy. Can you tell us a little bit about these plots and what we see here? Yeah, so this is more just showing the repetition values. So kind of what I was talking about earlier. So high repetition values would indicate that we're getting into kind of like degenerate loops, like repetitive loops. So where the model outputs the same thing over and over again, and I mean, we really see this in story generation for low values of k and n. Where Yeah, exactly there. So, you know, this is, these are like rep like repetition values of like point eight. So it's just like really just spitting out the same exact thing over and over again. And I mean, yeah, it's like, I think that looking at at this type of behavior in terms of information theory, it actually really makes, to me, it makes it makes sense why this is happening, right? If we're saying that we're always going to output the most likely word, like those are also the words that just have like no information content, right? And also, like, if I if I come to you, and I say, look, here is a sequence of words, it goes Apple, banana, peach, Apple, banana, peach, Apple, banana, and then to ask you like, what's next? I mean, it's quite likely that, you know, peach is the next thing. And that explains very well why if you keep repeating, you're sort of reinforcing even that that repetition, because as you keep repeating, the next repetition becomes more likely, yet the transmission of information is, is almost zero. Yeah. And but I mean, I think one thing that would actually be really interesting, one set of experiments that we have yet to run is to see, you know, if at the before you get into these repetitions, like if you start with with something, and then you like if you start with one phrase, and then go into typical sampling, right? Can you prevent some of these repetitive loops, because you've now come in with the objective that you want to transmit like more information on you don't want to be you don't want to transmit like a small amount of information, which is achieved by like doing by giving high probability low information words, right? So kind of seeing if typical sampling can almost help us break out of repetitive loops. Although by your own, by your own what you wrote, if you are, let's say in such a loop, or at the beginning of such a loop, the distribution would be extremely peaked, right? And at that point, typical sampling would also go for the for the high probability words, or is that I mean, and honestly, like, I think it should write like, at that point. But I mean, this is kind of why it's like before you get into the repetitions, right? So like, at that point, you know, where something like nuclear sampling might decide, like, yeah, like, the lowest information choice is, you know, just to repeat what's already been said. Yeah, if we can prevent, we can prevent those types of behaviors, just some small technicalities, whether where I want to ask you if you think that it's appropriate, do you think the absolute difference is an appropriate measure? Or why did you decide on that? That's the first thing. Second thing is, do you think this cutoff this hard, you know, I'm going to take this many words, and then I'm going to exclude the rest. And then I'm actually going to sample from that bunch of words, as if it were like the original distribute, like, with with their original logits. So just the technical implementation of the idea, what could be like, what are arbitrary choices? What are what are things that you did for a reason? And how could they be better? No, I think that's like a great question. Why absolute value versus, you know, square distance? And, and why the hard cutoff? I mean, to be honest, I think this was the original instantiation of the idea was, you know, just choosing words from like near the information content, near the expected information content. And I think, yeah, in order to really introduce this concept into the literature, it helped. At least what I thought was that it would help to have something that was akin to what most people are familiar with, which is nucleus and top case sampling, right? And so for better or worse, this method was kind of like, okay, here's something that's very parallel. That'll be easy to understand. You know, it's, it's, it's also just truncating the distribution, also like looking at the specific portion of the distribution. And that's where we'll sample from. Now, whether it's better to use the square distance, I mean, so we ran some additional experiments later on, like after releasing this draft, looking at things like the square distance, and, you know, trying to come up with a soft distribution. And yeah, they worked about like, about the same, sometimes a little bit like, honestly, I think I'm gonna have like, I think there's just a lot of research to be done here. I think there's a huge, huge body of research that can be done in sort of figuring out exactly what our objective should be. Perhaps learning this objective, like learning what the correct, what the correct formula right here should be. And that's, you know, that's to come in the future. So I can't say that square distance isn't better. Very well could be. All right. Is there anything else you want to get get rid of? How can can people get started with this? Is there code somewhere? There is code, right? I've seen that. Yeah. There's actually code in Hugging Face already. So if you have, I don't know if they've released a version since it entered the library. I mean, it's been in there for about a month now. So I think if you have, if you have the Transformers, the Hugging Face Transformers library installed from source, if you have pulled it in the last month, it'll be in there. And you know, when you generate, if you just add in the argument typical P equals something, then you'll have, you'll have typical sampling. And I mean, I really encourage people to play around with it. I mean, I, yeah, you know, you're, you're going to expect me to say this, but I've actually just been really impressed by the outputs of typical sampling. Just that they have been pretty high quality from my perspective. And interesting. Cool. Klara, thank you very much for coming here. And thank you. Thanks for the great conversation. Was a pleasure. You know, maybe you'll see another update on Archive with some of the things you've pointed out. Clean up some of my arguments. That would be, that would be excellent lore for the channel. Yeah. Cool. Thank you. All right. Thank you. It's Eye七.
 Hi, y'all. This is an interview with Clara Meister, who is the first author of the paper
 Typical Decoding for Natural Language Generation. This paper, I believe, is really important
 because it presents a new sampling method that makes language models output much more
 human-like texts. I've already made a review about the paper if you haven't seen that yet.
 Check it out. Clara has seen it and we're able to dive directly into the matter. This
 interview was very cool. I learned a lot. As always, if you like, leave a like, tell
 me what you think in the comments and I'll see you around. Bye bye. Hey there, today's
 sponsor is the course on Introduction to Graph Neural Networks. This is a course by my friend
 Zach Jost, who is an expert in graph neural networks. He's packed all his knowledge into
 one course that will educate you on both the theoretical and hands-on practical aspect
 on graph neural networks. Graph neural networks are really important. They're definitely one
 of the most interesting areas in deep learning right now. They've also powered a lot of recent
 advances in scientific breakthroughs, such as alpha fold protein structure predictions
 or better traffic predictions. If you use my link, you'll get a 15% discount on the
 course. Enrollment is open right now and lasts until April 1st or until spaces run out. All
 right, let's get into the video now. See you. Hello everyone. Today I'm here with Clara
 Meister, who is the first author of the paper, Typical Decoding for Natural Language Generation.
 Clara, welcome very much to the channel. Thank you. And thank you for having me. This was
 a really neat paper. I have to say I have just finished my last interview, not just
 now, but I finished my last interview about a system called BLEP. What they said is essentially
 they have a system that generates captions for images in an automated fashion. Then they
 have a filter that kind of weeds out the crappy captions. They use that as a means of generating
 more high quality data. They and many others before them have found that how you sample
 from a model, like from the language model they've trained, matters a lot. Specifically,
 they told me that nucleus sampling in their case was really a defining factor in getting
 more of a diverse sample set. They particularly compared it to greedy sampling and to beam
 search, which they found super underwhelming. I've come across a lot of systems in recent
 times, for example, AlphaCode as well. I don't know if you know how exactly AlphaCode does
 what it does. I don't either, but from the paper I could gather that they sample a lot
 of potential solutions and then they reduce those down by filtering and clustering. Again,
 they rely heavily on being able to sample diversely and to sample many, many different
 things. I've for a while now thought maybe our sampling objectives are wrong for certain
 applications, namely for the applications where we actually are interested in more of
 a diverse output rather than the most likely output. Along came your paper, which essentially
 exactly plays into this and suggests a new method. I was super happy to see this. I think
 it really hits a nerve of the time. If you would pitch it, like the elevator pitch for
 the paper, what would you say about it?
 Yeah, I would say that specifically for language generation, I think with these large models
 that we've been training, that when we're generating language from them, we need to
 take into account what we really want from the model, what our objective is. Also, what
 we just normally do when we're speaking, when we're writing, how we use language. Trying
 to think about having this, what these models are is essentially probability distributions
 over strings. That's kind of a strange concept. It's not probably how we imagine language
 in our heads. There is some evidence in psycholinguistics that that's kind of actually a pretty good
 metaphor for how language is represented in our head. How we then go from that to generating
 language and what the characteristics of the language that we typically generate are, I
 think we really want to take that into account when we're trying to generate language from
 these models. If you just ask me to say something randomly, what am I going to say? I'm probably
 going to say, I don't know. I don't really have these really common phrases. But if we
 want something more interesting, if you want me to say something more interesting, then
 I'm going to not just pull the most likely sentence out of thin air. I'm going to try
 to convey information in what I'm saying. I think that these models have sort of learned
 how to do that implicitly. We can ask them then to try and do this in a similar manner
 to how humans do. Yeah.
 So you pretty quickly get to this notion of typicality, which is a notion from information
 theory. You connect it to various disciplines in psycholinguistics. But a typical message
 as far as I can understand it is, well, as the name says, one that you would expect to
 see from sort of a communication apparatus. But it is, do I understand this correctly,
 is one that you expect to see if you assume that the communicators want to transmit the
 optimal amount of information? Is this the core assumption behind how we think about
 communication between humans?
 Yeah. One important thing is typicality in the context of communication channels is really
 only defined in the context of a message here, some sort of message that you're conditioning
 on and trying to convey. So in here, I mean, especially when you're sampling from a language
 model without having this implicit message that you're conditioning on in the background,
 I think it's kind of hard to really quantify what a typical message in natural language
 should be. And I think we're very careful to say that there is this nice intuitive link
 between typicality and how humans use language and what type of strings we might expect when
 using natural language. But there's a lot of aspects of human language that don't really
 fall into the paradigm that you can really apply typicality to.
 And so you inspire, let's say, by this notion of typicality, or you're inspired by. So you
 define the notion of a typical message, and that is sort of the average information content
 you would see. I made a bit of a characterization in my video. By the way, we have to inform
 the viewers that I use the old archive version, and you just updated it. And you corrected
 essentially all the little criticisms I had about notation and things like this, just
 to get the lore right. It wasn't me that caused it. You did it ahead. And then I used the
 old version.
 You know, props to you for picking them out. My advisor always says that every single paper
 out there pretty much has math errors in it.
 Oh, yeah. Don't worry.
 It takes a critical eye to find them. It's super easy to just glance over them, not realize
 them.
 Well, I think it was actually straightforward. The paper is really easily readable. So when
 we think about how humans communicate, and let's assume for a moment what you say that
 in your hypothesis here, any given word should have an information content close to the expected
 information content, i.e. the conditional entropy given prior context. In other words,
 we expect this difference to be small in human-like text. And you also say that the human goal
 over here is to transmit information effectively while also minimizing the risk of miscommunication.
 I made a bit of an example right here as if I explain math, or if I explain the chain
 rule to someone who does and does not understand math, is this an appropriate example? Is this
 an appropriate metaphor for what you're going for? Or is this totally off?
 No, I think in a way that's right. I think that's actually perhaps even more related
 to what we described later on, which is the rational speech act, which is how we also
 are taking into account the listener when we're forming our messages. That's definitely
 a component that's taken into account. So we'll modulate the amount of information that
 we are conveying to basically account for what the other person might know. And I think
 that you can kind of model that in different ways. You can say that, in your case, I think
 how you put it, I think is a totally valid way to see it. In that case, we can say that
 the information content for the speaker is going to be much higher than for someone else.
 So I mean, yeah, I think that's a good comparison.
 So this notion of the expected information content is pretty important here. And we say,
 okay, if I'm at a certain, let's say I've uttered half a sentence, and then I look at
 the distribution of the next word. And that distribution is just the distribution of the
 language itself, if I understand this correctly. So I have my training corpus, which supposedly
 is all of human language, I analyze it in my head, I determine what's the conditional
 probability for the next word in the training corpus. And then your claim is that what I
 do is I don't actually sample from that distribution, I'm going to adjust in inside of my head,
 the distribution that I sample from two, two words that closely match the expected information
 content. My question is, why, why do I do that? Like I see the problem with always picking
 the highest likely word, right? If I if I have a broad distribution like this, I don't
 want to do that. I don't want to just pick the most likely one. However, why can't I
 just sample from this distribution? It seems like enough times I would actually, you know,
 pick some other words that is also completely fine.
 Yeah, I mean, so first of all, I think one thing is, when we're forming language, we
 are, I mean, we arguably aren't like sampling from this distribution, right? We kind of
 know, I mean, maybe to some extent, we're sampling what we're going to say next. But
 I mean, I think the important thing to internalize is that we have a message that we want to
 convey right every time that we're using language. And the way that we choose to do that is like
 at a specific information rate, because we want to communicate efficiently. But we also
 want to make sure that our message gets across without like having to repeat ourselves or
 confuse someone or, you know, making them like spend an inordinate amount of time processing
 what we're saying. And so because of that, like we're not going to choose super low information
 words all the time, because that's just kind of inefficient.
 Yeah, like, I can I can say all these filler words, right with and still get across a message,
 but adding like, it's like that, you know, that person that takes forever to explain
 something just goes about it in a super, like slow and redundant way.
 Don't make fun of my videos.
 What are you talking about? So I think that's something to to think about. And then sorry,
 the second part of your question, I've already forgotten.
 I mean, I so I think I've what I've understood is that if we look at just the distribution
 of the next word, that is, in all of language that is across humanity, everyone who's uttered
 ever that first half of the sentence, this is the distribution of next word. However,
 when I consider that I actually have a message to convey, that distribution changes, right?
 Is that about the characterization of what, like, my question would be, why don't I just
 sample from this distribution right here, given that if you know, many words are possible,
 it will actually result in kind of a diverse sampling.
 Yeah, I mean, I think that you like, first of all, I actually do think that in the case
 of like a perfect language model that you could actually sample from this distribution
 and be fine. I think that there are some there are some artifacts that are a bit strange,
 like especially in models that aren't trained as well with like this this long tail distribution
 that like that tail isn't necessarily learned all the learned very well, like what those
 actual probabilities are. And so, you know, you end up with like, just oddities. And,
 but beyond that, I mean, I do think that, like, we're not. I mean, we are trying to
 modulate when we speak, like the amount of information that we have per word, right?
 To keep it even. And this is this is not I mean, this is something that is perhaps not
 very obvious, but it is something that's like well studied in psycholinguistics, like how
 we how we convey a message. And like the coding that we will use within natural language.
 And so, like, yeah, we we we take this into consideration when choosing the next word.
 Yeah, not to be too redundant or to be too surprising.
 Yeah, and to end, again, to transmit what we actually want to transmit, right? Because
 I have something that I want to say, and that means I can't just blindly sample from the
 distribution, I would never actually transmit what I wanted to say, would it be would it
 be possible that, let's say, if I could hypothetically determine, you know, what what kind of let's
 say I have a message I want to transmit, could I somehow define the information content of
 the next word, given the message I want to transmit, and maybe also given the sentence,
 you know, so far t smaller than or smaller than t.
 Well, that's, I mean, that's actually usually what we're we're doing. And in so in a task
 like abstractive summarization, which, you know, we see is something that we experiment
 with, we are conditioning on that message, essentially, you know, a message being the
 article, right. And so it is like, we are taking that into account when we're trying
 to build our next word. Yeah, and it is still like, this distribution should reflect the
 fact that there is a message that we want to convey. And, you know, given that message,
 it sort of, it sort of reflects that, you know, maybe this word that without that knowledge
 would have been very surprising. But like, with that knowledge, with knowing that, like,
 we want to transmit this message, actually, that word is like what we would expect. Yeah.
 Okay. My my question, what I'm trying to get at is, if I train my language model for abstractive
 summarization, right, the conditioning of the message is maybe already in not maybe
 in here, if I use a decoder only model, but like, my question is still, why is this distribution
 here not enough? Like, why, why do I need to cut out the most likely things? Even though,
 you know, sometimes I actually want to say them. So, I mean, I think it's just to be
 more human like. Yeah, that's okay. That's the most I can say is, yeah, it's a it's fine,
 it's fine. So, you you make you come up with and we're gonna, we're gonna go back to these
 plots because I find them super interesting as well. You define this typical sampling
 strategy where you say, okay, we we have this thing here, which is the expected information
 content of the next word. And then we're just trying to as closely as possible match that.
 So we're just going to select a subset of all the words that we could pick, which closely
 match that expected information content according to your hypothesis. And then we're going to
 sample according to the new distribution that only consists of the subset of these words.
 So in the video, I think I raised a point which is maybe more of a, I don't know if
 it's circular logic or a philosophical point. But all our training data, presumably of these
 language models comes from humans, you know, using language transmitting information. Therefore,
 right? Shouldn't like if I now train my language model, and I use your method to sample things,
 and you claim it's a human like way of sampling things, shouldn't that a result in the same
 distribution? And B, shouldn't it sort of the expected information content if I measure
 before and after, like if I measure it in the training corpus, and then if I measure
 it as an output of my model, shouldn't that be the same? Because presumably the training
 corpus is already generated from humans.
 I mean, yeah, I think like, yes, I think that makes sense if I'm understanding correctly.
 And I also think we're kind of seeing that like in the earlier plots, we're actually
 seeing that, like, if there is like an average amount of information, right, according to
 the model, there's an average amount of information that each word will contain. And I mean, human
 text seems to be coming from quite close to what the model has learned that average information
 rate to be.
 And do you, did you investigate the outputs of your model? And sorry, sort of redid those
 plots on the output of your model and observe the same, the same pattern?
 Yeah, so that's like, yeah, that's something we did as well. We looked at basically a few
 different decoding schemes and saw what the, what these distributions looked like for the
 outputs of those decoding schemes. And I mean, things like, you know, to nucleus sampling
 with like very popular, you know, popular values of P looked similar. And so did the
 ones from typical sampling. We didn't, I think, honestly, they do look, they by visual, like
 visually, they look pretty similar, which is nice. It's also nice to see that sort of
 these more, these vetted decoding processes that have like stood the test of time are
 also actually mimicking these distributions. I think that if we wanted to be robust about
 it, we'd probably want to, you know, come up with some sort of quantification for how
 different these distributions are. And use that perhaps to see if that correlates with
 how well these decoding methods perform in terms of things like human evaluations.
 So can you tell us the story behind these plots a little bit more? Because you define
 epsilon in terms of an absolute value yet here I see values that are less than zero
 to both sides. So I didn't know which one is which. What's epsilon here?
 I tried to make it clear in the caption of the text, but I don't think I did.
 I mean, if I guess correctly, it's the conditional, it's the expectation minus the actual information.
 No, so it's actual information minus...
 I would have gotten it wrong.
 Oh, wait. No, no, I think you're right. No, no.
 Maybe you can tell us what does it, because these are kind of, so it's more, if I see
 this correctly, more sort of mass on the left side of these close to this boundary, which
 is really interesting. And then there's a long tail on the right hand side. What does
 that tell us about human language?
 I mean, that's like a very deep question. And I'm not entirely sure about what the shape
 of this distribution means. And I think it's very interesting that this is the shape of
 the distribution. And actually, we used a few models here, and all of them kind of did
 look like this, where you had this peak and then sort of a long tail. And yeah, I mean,
 I think that that's an investigation in its own right about how humans use language.
 So yeah, by the way, it is information content minus entropy. So remember, so low information
 content, high probability, right? So actually, human language tends to be to the like on
 the higher probability side of conditional entropy.
 This thing right here. So if we if we're way out on the right, it means that we actually
 transmit a lot of information actually more than would be expected. So there is it doesn't
 that there is a long tail of very high information words, let's say, do you think so because
 you in one thing that I skipped over that in the video review, but you make this point
 of humans, what they probably do is they want to everywhere in the message, they want to
 have kind of a constant information rate. So every word should approximately transmit
 this this expected information. So as you go through the sentence, do you think this
 could be violated a little bit because humans, most of them do tend to have like a short
 term memory of three to four words or so that they, you know, can keep keep ready in the
 sentence, maybe I can transmit this super high information word. And then before my
 receiver gets super confused, I can follow that up with like two or three clarifications,
 which which would be then maybe here in the lower information content, but they would
 be more. Yeah, I mean, so, like, I think it's hard to always avoid moments of high information.
 I mean, for example, if you're giving if you think about this very literally, in terms
 of like what those words could be, you know, they could be like someone's name, right.
 And that's kind of like you're introducing someone that's always kind of going to be
 like a high information moment, right. You have to remember, I mean, we always forget
 people's name, people's names, obviously, there's like, there must be a lot of information
 in those names. So very off the cuff explanation. But I mean, yeah, so I think it is hard to
 just 100% of the time, avoid those instances. But I mean, this is talking about sort of
 on average, what we're doing when we're constructing language. And I mean, so I guess I couldn't
 say whether in those moments, we want we try to perhaps on either side, balance out, like
 with lower information words, this high information word, because I mean, you know, maybe maybe
 we do in order to give the listener some time to internalize this information. But there
 are also especially with with speaking, which is a different domain than writing, right,
 there are other ways that we can modulate high information words, right. So we can elongate
 our speech to basically spread out information over time, right. And so it's not like here,
 we're just evaluating text. So, you know, we, I think, especially in text, we're going
 to see these longer tails, because you can't sort of distribute information over too many
 words in certain cases, like in the case of introducing a name. Yeah, I think that's
 and also it has to be said that, you know, you can, if you go to the left, you get into
 the super low information words. And there is only that many of them, right? As soon
 as I'm at the and, uh, right there, there aren't that many. However, there is, in fact,
 a long tail just in the language of super high information words that are quite unlikely.
 So maybe that plays a role into it as well. About these plots, you say you draw two, two
 different conclusions right here, which the first one is the peak nature reveals that
 humans indeed tend to form language with per word information content quite close to their
 expected information content. So this is kind of, you know, here is data that shows our
 hypothesis is correct. And the second one is the centering of these distributions around
 a value close to zero reveals that our probabilistic language generators are learning what this
 rate is, which, and my point was a bit when in order to make point one, you need point
 two as an assumption, right? You need, you need to claim, well, I can only say this because
 I assume our language models are modeling the probabilities of language well enough.
 Otherwise I could not conclude point one. Likewise, you couldn't conclude point two
 without having point one as an assumption. Is this, am I overlooking something here?
 Well, so, I mean, I think the point here that we wanted to get across was really that, you
 know, two things should be looked at in these graphs, which is the centering of the graph
 and also the shape of the graph. And I mean, so I think there is, there is an assumption
 that kind of has to be made here. I don't think it's as quite as severe as, as what
 you've mentioned, but I mean, it is sort of that this enter, this information rate is
 kind of a ground truth of sorts. But I mean, you know, you could, for example, shift, like
 you could shift to that entropy rate. You could shift the entire distribution and still,
 you could shift H and all the P's and you know, all of, all those numbers and still
 technically get the same distribution. So that I agree with. But like, I mean, I think
 like looking at the peakiness of it, clearly we're seeing that, you know, humans are generating
 language around a certain...
 Something, right?
...content.
 Yeah. Yeah.
 What if it were centered around two instead of zero, right? It would be as peaky.
 Well, yeah, I mean, yeah, as peaky then like, yeah, like we'd probably be, that'd probably
 show that humans communicate at like a very low information rate, right? Or, yeah. So,
 but no, I mean, it's around, like it does seem to be close to this expected information
 rate. And I think one other, like the part two is really trying to show that like there's
 this, we would expect that if our model understands that, you know, humans are speaking at around
 an average information rate, that this distribution would be centered around, like on average,
 it would be predicting that information rate for a given word or like that information
 content, that probability for a given word. And it does seem to be doing this.
 Cool. Yeah, this is just a bit of a nitpick for me. I'm totally on board with, I mean,
 it's pretty clear the language models do model these probabilities relatively correctly,
 especially the ones with the higher probabilities. And I'm fairly convinced by these plots that
 what you're doing is something sensible.
 Yeah, no, I mean, I think you bring up a really important point. And I actually, like I'd
 spent a long time thinking about whether or not it was too circular, like, you know, whether
 you could have one without the other, really. And I mean, I think, like, I think at some
 point I came up with some examples, like some counterfactual examples where actually you
 could have one without the other. And of course, now, like, I can't remember what they are.
 But yeah, it's, it's, it's, I think, I think people understand what you're, what you're
 saying.
 There's definitely like a degree of freedom there, right? There's definitely something
 that could change that, you know, you could get those same results. And I think, but I
 think, like, that thing that could change would be whether the information rate learned
 by the model is like the quote, human information rate, the actual human information rate. And
 I'm actually not entirely sure that's important. It just has to be, it just has to get it right,
 like relative to what it's predicting the probabilities for words, right?
 Do you want to tell us a little bit about the experimental results? Because I have not
 gone into these at all during the paper review, things that you would like to highlight or
 anything like that?
 Yeah. So, like, as Yannick mentioned, there's a new version on archive, where we are, we
 also present a few different values for nucleus and top K, as in like the same, you know,
 same number of values.
 Oh, yeah, the hyperparameters. Sorry about that.
 No, no, I think it's very reasonable. I mean, the thing is, like, you know, there were only
 so many human evaluations we could afford. And we thought, like, you know, we should
 probably test out more values of our own method, since no one has done this before. But like,
 a lot of people have looked at nucleus and top K sampling. But then once it seemed like,
 okay, this is worth, this is research worth doing, we were able to get a little more money
 and launch a larger human evaluation. So those results are now in the paper. I mean, I think
 one thing that was really interesting for us was actually just the variety of values
 of tau that worked well. I mean, basically, like, most values of tau worked well. There
 wasn't like a huge difference between all of them, which we thought was really cool,
 because in comparison to nucleus and top K sampling, those methods were really dependent
 on N and K. And I mean, I think there was like a little, like, if you just look at the
 output of these models, you know, if you have a large tau, then maybe qualitatively, you
 could say that the text is like a little more normal, like a little more standard, and then
 maybe a little more diverse for low values of tau. But I mean, basically, it was just
 for, it was just interesting to see that for these two tasks, at least, that, you know,
 variety, like it wasn't, you didn't really need to tune tau that much, just kind of,
 kind of worked.
 It's important, right? Because that's one of the issues with these things is that if
 I have to tune the thing to every new task I do, I'm a lot less certain in, you know,
 kind of the generalization of this even within the same domain. But if it's interesting to
 hear and if it's really a kind of a handle on the craziness that I get out of these models,
 that could actually be even a cool property, right? If you say, actually, most values work,
 but it is, you know, it changes just the style. I think that that is a useful hyperparameter
 rather than a nuisance like in nuclear sampling. You know, if I don't get it right, it's going
 to be crap.
 Yeah, well, I would like to think that that's the case. I'm slightly biased here.
 Yeah, is there any, I mean, you run various automated tests in abstractive summarization
 and story generation. Most of the time, the typical sampling is on top of the pack, sometimes
 not, especially here in the story generation on some of these automated evaluations. Is
 that kind of an interplay between the evaluation, how the evaluation is done and the methods?
 Or if that is that a property of the task itself? What can you tell us about this?
 I mean, so I think a lot of these metrics, I think a lot of these metrics can only tell
 us so much. And, you know, the text that we end up generating, how it performs in terms
 of these metrics, I think like you'll see, for example, in human text, you'll get reasonably
 different values. Like you can get reasonably different values for things like repetitions
 within reason and the text be equally as good, at least qualitatively. So like, I think the
 important, I don't know if it's important is the correct word, but one of the critical
 things for us was like looking at whether we could avoid this really degenerate behavior
 with models. Because I think that's something that's like one of the bigger problems in
 language generation is just like this tendency for these methods to fall into repetitive
 loops. And I mean, we basically just like, we didn't really see any of that in using
 our method. And so I think that was an important takeaway. So yeah, I mean, always kind of
 performing well in terms of this, in these metrics that show how repetitive or redundant
 text is. I think it is what we would expect, right? You know, we're saying that like if
 text is, we want text to be about as redundant as human text is, because that's like one
 metric you can use to quantify information content, right? So it was good to see that
 that like, at least, it's a necessary, not sufficient criteria, but it was good to see
 that it was met.
 Yeah, I was just looking, like just now looking at perplexity, and yours is in bold. And I
 was like, wait a minute, lower perplexity is better usually. But then I realized what
 you have to do here is obviously match the perplexity of the reference text as closely
 as possible. So the goal is to be as close as possible to that number, which is really
 astonishing to see because in machine translation, people are fighting for 0.1 perplexity or
 so for the new state of the art. And here it's a difference of, it's quite a magnitude
 of difference between these methods, which is cool to see. And I think shows quite well
 that in something like story generation, these models might really just not, overfit is the
 wrong word, but overproduce not as creative outputs, or maybe even degenerate ones, as
 you say.
 I mean, I think actually in the context of machine translation, and this is something
 that an experiment that I want to personally perform is look at what the average perplexity
 of the reference text is, right? I mean, so and the generations, right? So the one thing
 about machine translation is typically we're evaluating on things like blue, right? Not
 perplexity so much that we're evaluating on the generations themselves, rather than the
 evaluation of the reference text, like what the perplexities are. But I mean, it would
 be, to me, it would be interesting to see what the perplexity of good generated text
 is compared to human like text. And I think in that case, they would actually probably
 both be quite small. At least that's my intuition. Of course, one artifact that I think would
 kind of get in the way of these experiments is the fact that machine translation often
 uses label smoothing, right? And label smoothing is basically like a form of entropy regularization.
 So it makes these distributions higher entropy even if they shouldn't be. And that actually,
 I mean, basically, you can read other papers about this that will explain it. But it is
 kind of it does interact with beam search. It's like the match of beam search plus label
 smoothing tends to work quite well. But I think if you were to really perform these
 types of experiments to understand what the types of perplexities for machine translate,
 like for translations, good translations would be, I think, yeah, you'd need to do it with
 a model that doesn't that hasn't had this sort of artificial inflation and entropy.
 Do you think our training objectives are the correct ones? Let's think of something like
 story generation is pretty, because what I'm hearing now is that, well, label smoothing
 but plus beam search works, but it's more like a hack to get around the weaknesses of
 beam search without label smoothing. Do you? And that is, you know, something I can maybe,
 you know, get get behind. Do you think we have the correct training objectives if our
 goal is really to create diverse and interesting set of outputs? Do you think it's a good strategy
 to train, let's say maximum likelihood, and then sample using something like typical sampling?
 Or should we also change our training strategy?
 So I personally think that maximum likelihood is a pretty robust objective. I mean, in terms
 of like the information theory perspective, I mean, when you when you are maximizing likelihood,
 right, you're also minimizing KL divergence. So you are basically looking for the model
 that assigns the same information contents to strings as as the empirical distribution.
 Right. So it's like they're just equivalent. And so I think if you take that into account,
 basically, if you take into account exactly what you're doing with your objective, and
 then from that, you know, go on to, okay, well, given given this distribution, right,
 how how would we go about how would like we as humans go about generating from this distribution?
 Or you know, how would if like you're generating an image, like how would nature go about like
 generating from this distribution? I think, you know, it's really important to I don't
 think there's a correct way necessarily to go about training and decoding. But I think
 we really need to take into account more their interaction and understand like, what is going
 on within that interaction.
 Yeah, I mean, I'm all on board, because it also means that we can use we can reuse the
 same model for multiple, let's say tasks, if we swap out our decoding strategy. Can
 you tell us a little bit about these plots and what we see here?
 Yeah, so this is more just showing the repetition values. So kind of what I was talking about
 earlier. So high repetition values would indicate that we're getting into kind of like degenerate
 loops, like repetitive loops. So where the model outputs the same thing over and over
 again, and I mean, we really see this in story generation for low values of k and n. Where
 Yeah, exactly there. So, you know, this is, these are like rep like repetition values
 of like point eight. So it's just like really just spitting out the same exact thing over
 and over again. And I mean, yeah, it's like, I think that looking at at this type of behavior
 in terms of information theory, it actually really makes, to me, it makes it makes sense
 why this is happening, right? If we're saying that we're always going to output the most
 likely word, like those are also the words that just have like no information content,
 right?
 And also, like, if I if I come to you, and I say, look, here is a sequence of words,
 it goes Apple, banana, peach, Apple, banana, peach, Apple, banana, and then to ask you
 like, what's next? I mean, it's quite likely that, you know, peach is the next thing. And
 that explains very well why if you keep repeating, you're sort of reinforcing even that that
 repetition, because as you keep repeating, the next repetition becomes more likely, yet
 the transmission of information is, is almost zero.
 Yeah. And but I mean, I think one thing that would actually be really interesting, one
 set of experiments that we have yet to run is to see, you know, if at the before you
 get into these repetitions, like if you start with with something, and then you like if
 you start with one phrase, and then go into typical sampling, right? Can you prevent some
 of these repetitive loops, because you've now come in with the objective that you want
 to transmit like more information on you don't want to be you don't want to transmit like
 a small amount of information, which is achieved by like doing by giving high probability low
 information words, right? So kind of seeing if typical sampling can almost help us break
 out of repetitive loops.
 Although by your own, by your own what you wrote, if you are, let's say in such a loop,
 or at the beginning of such a loop, the distribution would be extremely peaked, right? And at that
 point, typical sampling would also go for the for the high probability words, or is
 that
 I mean, and honestly, like, I think it should write like, at that point. But I mean, this
 is kind of why it's like before you get into the repetitions, right? So like, at that point,
 you know, where something like nuclear sampling might decide, like, yeah, like, the lowest
 information choice is, you know, just to repeat what's already been said. Yeah, if we can
 prevent, we can prevent those types of behaviors,
 just some small technicalities, whether where I want to ask you if you think that it's appropriate,
 do you think the absolute difference is an appropriate measure? Or why did you decide
 on that? That's the first thing. Second thing is, do you think this cutoff this hard, you
 know, I'm going to take this many words, and then I'm going to exclude the rest. And then
 I'm actually going to sample from that bunch of words, as if it were like the original
 distribute, like, with with their original logits. So just the technical implementation
 of the idea, what could be like, what are arbitrary choices? What are what are things
 that you did for a reason? And how could they be better?
 No, I think that's like a great question. Why absolute value versus, you know, square
 distance? And, and why the hard cutoff? I mean, to be honest, I think this was the original
 instantiation of the idea was, you know, just choosing words from like near the information
 content, near the expected information content. And I think, yeah, in order to really introduce
 this concept into the literature, it helped. At least what I thought was that it would
 help to have something that was akin to what most people are familiar with, which is nucleus
 and top case sampling, right? And so for better or worse, this method was kind of like, okay,
 here's something that's very parallel. That'll be easy to understand. You know, it's, it's,
 it's also just truncating the distribution, also like looking at the specific portion
 of the distribution. And that's where we'll sample from. Now, whether it's better to use
 the square distance, I mean, so we ran some additional experiments later on, like after
 releasing this draft, looking at things like the square distance, and, you know, trying
 to come up with a soft distribution. And yeah, they worked about like, about the same, sometimes
 a little bit like, honestly, I think I'm gonna have like, I think there's just a lot of research
 to be done here. I think there's a huge, huge body of research that can be done in sort
 of figuring out exactly what our objective should be. Perhaps learning this objective,
 like learning what the correct, what the correct formula right here should be. And that's,
 you know, that's to come in the future. So I can't say that square distance isn't better.
 Very well could be.
 All right. Is there anything else you want to get get rid of? How can can people get
 started with this? Is there code somewhere? There is code, right? I've seen that.
 Yeah. There's actually code in Hugging Face already. So if you have, I don't know if they've
 released a version since it entered the library. I mean, it's been in there for about a month
 now. So I think if you have, if you have the Transformers, the Hugging Face Transformers
 library installed from source, if you have pulled it in the last month, it'll be in there.
 And you know, when you generate, if you just add in the argument typical P equals something,
 then you'll have, you'll have typical sampling. And I mean, I really encourage people to play
 around with it. I mean, I, yeah, you know, you're, you're going to expect me to say this,
 but I've actually just been really impressed by the outputs of typical sampling. Just that
 they have been pretty high quality from my perspective. And interesting.
 Cool. Klara, thank you very much for coming here.
 And thank you. Thanks for the great conversation.
 Was a pleasure.
 You know, maybe you'll see another update on Archive with some of the things you've
 pointed out. Clean up some of my arguments.
 That would be, that would be excellent lore for the channel.
 Yeah.
 Cool. Thank you.
 All right. Thank you.
 It's Eye七.