GPT-2: Language Models are Unsupervised Multitask Learners
gpt2 transformer language model deep learning nlp openai security translation neural network attention attention mechanism unsupervised learning controversy
A look at OpenAI's new GPT-2 model and the surrounding controversy.

https://blog.openai.com/better-language-models/

Abstract:
Natural language processing tasks, such as question answering, machine translation, reading comprehension, and summarization, are typically approached with supervised learning on taskspecific datasets. We demonstrate that language models begin to learn these tasks without any explicit supervision when trained on a new dataset of millions of webpages called WebText. When conditioned on a document plus questions, the answers generated by the language model reach 55 F1 on the CoQA dataset - matching or exceeding the performance of 3 out of 4 baseline systems without using the 127,000+ training examples. The capacity of the language model is essential to the success of zero-shot task transfer and increasing it improves performance in a log-linear fashion across tasks. Our largest model, GPT-2, is a 1.5B parameter Transformer that achieves state of the art results on 7 out of 8 tested language modeling datasets in a zero-shot setting but still underfits WebText. Samples from the model reflect these improvements and contain coherent paragraphs of text. These findings suggest a promising path towards building language processing systems which learn to perform tasks from their naturally occurring demonstrations.

Authors:
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever
 Hi, today we're looking at language models are unsupervised multitask learners by Alec Radford, Jeffrey Wu, Reverend Child, David Luan, Dario Amadai and Ilya Sotskyver from OpenAI. This paper has generated a bit of hype in the last few days, so I wanted to go over it, basically take a look and take a look at the surrounding, let's say controversy. So let's actually have a look at the blog post that OpenAI released along with this paper. They say, we've trained a large scale unsupervised language model which generates coherent paragraphs of text, achieves state of the art performance on many language modeling benchmarks and performs rudimentary reading comprehension, machine translation, question answering and summarization all without task specific training. So this sounds quite suspicious at the beginning, but we're actually going to have to look at how they do this. It sounds really good being able to do a rudimentary translation without any training on translation itself, just learning a language model. But this has been continuing a trend in recent kind of time where we see that the better your language model gets, the better basically your model gets on all these kind of language tasks. Alright, they go into this and we'll see how they do it. So basically what they've done is they've taken kind of a bigger dataset of language model, of language model dataset, which is about 40 gigabytes of internet text, I say this is here on the top. So it's one of the largest kind of text datasets there is unsupervised. And they also taken one of the largest language models. So they have their largest transformer based model has 1.5 billion parameters. So they take huge amount of data, huge model, they train this on, they train the model on the data and what comes out is like giant super language model that is able to perform all these cool tasks. So here they have like a bit of a sample. So what they can do is they can basically, so the way a language model works is you always try to predict the next word based on what you've already seen. So you kind of query it by giving it some starting words and it needs to continue the text from there. So here system prompt on top you see in a shocking finding scientists discovered a herd of unicorns living in a remote previously unexplored valley in the Andes mountains. Even more surprising to the researcher the fact that the unicorns spoke perfect English. And then the model continues. The scientists named their population the population after their distinctive horn, Ovitz unicorn. These four horns silver white unicorns were previously unknown to science. Now after almost two centuries the mystery of what sparked this odd phenomenon is finally solved. I mean you can even read this it's really, really coherent text and it's quite surprising. I think it's like slightly cherry picked but still the fact that a model is able to do this is unprecedented. Especially like since it's like a general language model not specific to the task of continuing news articles about unicorns or anything. So yeah they go into these findings we'll see them in the paper and they also say that yeah they can now do all these kind of tasks like question answering reading comprehension in a zero-shot fashion. So at the end here they say what it's capable of. So let's say AI writing assistants more capable dialogue agents unsupervised translation blah blah blah. They also say a few kind of let's say bad implications. Generate misleading news articles impersonate others online automate the production of abusive or fake content to post on social media automate the production of spam or phishing content. They liken it to a system called deep fakes which generates really well crafted let's say videos of people. So that the kind of they frame it in a way as this could be used for dangerous things and they say they aren't releasing they're only releasing the small version of GPT-2 along with the code. They're not releasing the data set training code or the GPT-2 this is the big model the model of weights right. And they do this they cite safety concerns. So I mean the community basically is going nuts over this statement this decision not release basically the code or the model or the data set to the world. And so if you search on Twitter for GPT-2 then everyone basically has an opinion of whether or not this is a good thing or not apart from people testing it out. So they've given access to like a selected set of journalists to an API where they can query the model. So there are some samples flying around but basically people are just debating whether or not this is a good thing or not and I mean it's just hilarious to go along with this and to just read all the people having opinions. I mean I've given my opinion as well just chime in it's a fun ride especially like this post here on reddit machine learning says should I release my NIST model or keep it closed source fearing malicious use. Today I trained a 23,000 layer ResNet got a 99% accuracy and MNIST. I'd love to share the model but I fear it being used maliciously. What if it is used to read documents by the Russians? What are your thoughts? I mean yeah this is in essence I mean in essence it's that right. It's like yeah come on. So I can just give my opinion up front. I think a lot of things came together here and I think that this being OpenAI being kind of this initiative I mean it's never been there before they're not an academic institution. They're not a company but still they you know they're researchers they want to have a career so there's lots of pressures on them. There's pressure to further publish so I think that that's kind of an underlying motivation to not release your model and your code and your data set is you actually you know there's a lot of work in this and you actually might want to get more than one paper out of it so keeping these things to yourself is basically a surefire guarantee you're going to get another two, three, four, five papers out of this data or model or whatever. It's also a good way to kind of generate press if you basically say oh we're not releasing but we have this really good model and there's one thing on Twitter right I mean you can't probably can't find it but says like step one my model is state of the art step two my model is state of the art but generalizes better to other tasks step three my model does the same thing but with fewer parameters and step four my model is so good I can't even talk about it. So basically I think a lot of things came together this press generating the pressure to create more kind of papers out of it and genuinely security concerns. I think being open AI and open AI kind of was established as a way to let's say the demo like their statutes pretty clearly say we want to open AI and research it in ethical use and you have backers like Elon Musk that talk all the time about kind of safety related issues in AI. I think there's a lot of pressure on these people to with everything they do basically have an ethical component. So everything they do is kind of scrutiny to this does this have ethical implications and where can we basically stand out from the rest of the community by doing something it doesn't need to be more ethical just needs to be different with an ethical reason behind reasoning behind it and this I think this is it I think there's a lot of things coming together I don't I don't think anyone like maliciously thought oh this you know we're gonna do this it's gonna generate so much press or I don't think anyone actively thought ah we'll just keep it secret we're gonna get so much more papers out of it I honestly think that the reasoning was there's a lot of you know a lot of pressure to do this ethical things when there's there's not if you think about it it's yeah it's a good language model and it can generate text but you can also hire you know people to generate text to generate fake news to do phishing and spam it's just a bit more efficient now right and yeah so it's it's unprecedented that you don't you don't release this research kind of cold war style so it's not really dangerous to release this and it's just delaying the inevitable basically but I think that the pressure the pressure was on and they made a decision that they thought was in line with their values and I think the this just neatly aligns with the underlying the other benefits to them that yeah all right so let's dive into the paper the paper is actually not you know too too much content there what they basically say so far is that a lot of a lot of these kind of papers on these tasks they they say the dominant approach to creating ML systems is to collect a data set of training examples demonstrate correct behavior train a system to imitate test its performance on in IID samples so they basically say the there's kind of the single task training on single domain data sets is a major contributor to the lack of generalization observed in current systems so they basically say these language systems they don't generalize like a QA system might be trained on a QA task but it you know has nothing to do with the task is basically a little bit different and even in multitask learning they say multitask learning is a promising framework but also it's kind of say it's still nascent and there's only very few different tasks right to do so they basically say you need basically a big big unsupervised model and it will implicitly learn all of the kind of special tasks and yeah so they say there there are approaches that basically basically learn these language models but then still require supervised training so basically fine-tuning this has been the this is for example the bird paper we discussed in the in the last video or two two videos ago that learns a giant language model but then does fine-tuning for each of these tasks and gets really well what they want to do here is basically simply learn a language model and then investigate whether or not the language model can perform downstream tasks in a zero-shot setting that means without any parameter or architecture modification so no fine-tuning all right so what they do so basically what a language model is if for those who don't know it's it's if you have a sequence of text let's say a b c d e these are words let's act like some actual words the cat sat on the mat and so on and you and you a language model is you kind of remove the end of the sentence at some point and ask the model what comes next right that's a language model I mean there's different kinds of language models specific language ones but that's the basic the basic thing so the you just ask the model what's next so you can you can do a lot of unsupervised training because you don't need a label data set for this you simply need a text corpus and that's basically all they do they use transformers which we've also discussed in attention is all you need paper so if you if you don't know what transformers are go back and look at that yeah all right so basically they say a lot of these special tasks like translation and question answering can be framed in language model way for example if you simply input if this is your text translate to French and then the English text and then the French text right and then at at test time basically you leave away the French text you simply ask the language model what comes next right if and its input is translate to French and then English text this is the translation framed as a language model task because you can specify the task that the language allows to do also as language so this is quite this is quite an interesting approach and one they exploit here and they basically say well since in a large and diverse corpus of web pages that they collect here some there is going to be some websites that basically all do translation from English to French and the model can learn from that so here in this paragraph they basically list examples of naturally occurring demonstrations of English to French and French to English translation found throughout the training data set so basically this is this is how the model could learn let's just look at one I hate the word perfume Bursas it's somewhat better in French right so there's a way in just an unsupervised setting where the language model could learn right if you just cross out this word at the end and you just ask the model what comes next right the model sees I hate the word perfume Bursas it's somewhat better in French period colon then the model has to put something there and the most logical continuation is to put the French word for perfume right so that that's kind of how they frame translation and these other tasks in language model way all right so they talk about the training data set which is a major component here they say they make a new training data set because all of the current ones aren't sufficient they say most prominent source of diverse nearly unlimited text is web scripts such as common crawl while these archives are many orders of magnitude larger than current language modeling datasets they have significant data quality issues so to say content are mostly unintelligible and so on so they basically describe here how they scrape a new web scrape which emphasizes document quality they go on reddit basically and scrape all outbound links from reddit that have received at least three karma which means that it yeah three up votes for a post of a link which basically means that three humans agreed that this was a good link so so they that's that's how they collect the data set resulting data set web stack web text contains text subset of the 45 million links they then kind of clean this and scrape it down and remove some stuff and they they end up with a large corpus right and then they talk about how they represent the input which is byte pair encoding style it's not exactly by parent coding it's a byte pair encoding inspired encoding we won't make a video about this by itself because it's really interesting but basically it's you can think of it as tokenization and pre-processing right then they say they they show their models so architecture hyperparameters basically these are these are their models this is the smallest one this second smallest one they say it's the same size as BERT so the the language model by google that we've looked at and then the largest one 1.5 billion parameters I mean that's huge and yeah they say it's ten times larger than the previous so the first one is their previous model and this now is this is the GPT-2 model that that gets all these these nice results so they do experiments first they do experiments on language modeling itself right so they train on their on their corpus and then they evaluate on a bunch of other language modeling corpus so these up here are language modeling corpuses and the state of the art is in this row and then you just look at basically the bottom row compare to their largest model this this is perplexity where it says PPL and the I think this here is is is accuracy so perplexity lower is better which you can you can see here the previous state of the art was 39 on wiki text 2 they get to 18 with accuracy obviously higher is better so the the kind of previous accuracy in Lombarda was 59 they get to 63 basically improve everything except for this 1 billion word corpus and they they also explain why they say this is the most heavily pre-processed text and so on so that basically they basically are really good at language modeling even though they train on a different data set that's the the point right the point is they train on their own corpus and then they go and just evaluate on the test set of these of these new of these new tasks and they become better basically than the models that trained on the training data set of that particular task all right so they they do a number of further experiments where they basically show that the model has now learned kind of implicitly learned a number of different tasks so let's look at for example summarization this just want to show an example of how you can do this so summarization summarization task is you get a long text you need to produce a short text and that short text is then compared to short texts that humans wrote when the task was to summarize the long text and you get points on how much your text overlaps with these human texts all right so they they say we test gpt2's ability to perform summarization on the cnn and daily mail data set to induce summarization here's what i found interesting we add the text tldr after the article and generate 100 tokens right then they say they need to reduce repetition and so on but basically this this this is right this is the way you can frame summarization by text input so i find this just kind of a really nice way to think about these problems the fact that instructions of the task can be given as text this is a very nice example here so basically you you put you as input you put the entire article right and so you here is the the cnn article blah blah blah it's super long right and then here you put tldr which is for those who don't know it's too long didn't read so people use this this phrase to indicate that then they will write a short summary of whatever was before here they will either put this at the beginning or at the end of a long text right to to say to people okay if you if you don't want to read all this just read this down here um gives you the gist of it which is exactly summarization so if you then take this away and ask the language model what's here right basically throughout the training corpus the language model will have encountered such pieces of text with a tldr in it and the language model might have learned that whatever is down here is a short version of whatever is up here and thereby if you then ask the language model what comes next here right the language model might learn aha i need to summarize whatever is above and that's the my best shot at completing at at answering the question what comes next and yeah so they get you know surprisingly good results um from from this so they say on the commonly reported rouge 12l metrics the generated summaries only begin to approach the performance of classic neural baselines just barely outperforms selecting three random sentences from the article uh but but um still it it um while qualitatively the generations resemble summaries they often focus on recent content from there to color confuse specific details so this is kind of a task where it kind of worked but not really um but i just find it it's really interesting that that it it kind of how they frame the task and how this can still so it still kind of works and that's the the gist here in all of these tasks is also with like translation they obviously don't get near the performance of a system specifically trained to do this task but they all also always say it kind of works right it's sort of sort of it learns something and their entire point of this paper is to say well look um yeah the the the diversity of tasks the model is able to perform and i would say kind of perform in a zero shot setting suggests that high capacity models trained to maximize the likelihood of a sufficiently varied text corpus begin to learn how to perform a surprising amount of tasks without the need for explicit supervision so yeah their entire point is if we train on such varied data that kind of um that spans the entire range of human language expression the the kind of tasks we want these systems to do will be learned implicitly so basically it points to let's get an even bigger corpus let's get even bigger models and we might get even better unsupervised zero shot way in these kind of special tasks and general language understanding all right so that that was basically i've jumped over a lot of points but i encourage you to look into this to look into the specific experiments they're really interesting the way how they framed things and um give just just shout your opinion around about whether or not the publishing is a good thing or not it's really funny i love it um and with that have a good day
 Hi, today we're looking at language models are unsupervised multitask learners by Alec
 Radford, Jeffrey Wu, Reverend Child, David Luan, Dario Amadai and Ilya Sotskyver from
 OpenAI. This paper has generated a bit of hype in the last few days, so I wanted to
 go over it, basically take a look and take a look at the surrounding, let's say controversy.
 So let's actually have a look at the blog post that OpenAI released along with this
 paper. They say, we've trained a large scale unsupervised language model which generates
 coherent paragraphs of text, achieves state of the art performance on many language modeling
 benchmarks and performs rudimentary reading comprehension, machine translation, question
 answering and summarization all without task specific training. So this sounds quite suspicious
 at the beginning, but we're actually going to have to look at how they do this. It sounds
 really good being able to do a rudimentary translation without any training on translation
 itself, just learning a language model. But this has been continuing a trend in recent
 kind of time where we see that the better your language model gets, the better basically
 your model gets on all these kind of language tasks. Alright, they go into this and we'll
 see how they do it. So basically what they've done is they've taken kind of a bigger dataset
 of language model, of language model dataset, which is about 40 gigabytes of internet text,
 I say this is here on the top. So it's one of the largest kind of text datasets there
 is unsupervised. And they also taken one of the largest language models. So they have
 their largest transformer based model has 1.5 billion parameters. So they take huge
 amount of data, huge model, they train this on, they train the model on the data and what
 comes out is like giant super language model that is able to perform all these cool tasks.
 So here they have like a bit of a sample. So what they can do is they can basically,
 so the way a language model works is you always try to predict the next word based on what
 you've already seen. So you kind of query it by giving it some starting words and it
 needs to continue the text from there. So here system prompt on top you see in a shocking
 finding scientists discovered a herd of unicorns living in a remote previously unexplored valley
 in the Andes mountains. Even more surprising to the researcher the fact that the unicorns
 spoke perfect English. And then the model continues. The scientists named their population
 the population after their distinctive horn, Ovitz unicorn. These four horns silver white
 unicorns were previously unknown to science. Now after almost two centuries the mystery
 of what sparked this odd phenomenon is finally solved. I mean you can even read this it's
 really, really coherent text and it's quite surprising. I think it's like slightly cherry
 picked but still the fact that a model is able to do this is unprecedented. Especially
 like since it's like a general language model not specific to the task of continuing news
 articles about unicorns or anything. So yeah they go into these findings we'll see them
 in the paper and they also say that yeah they can now do all these kind of tasks like question
 answering reading comprehension in a zero-shot fashion. So at the end here they say what
 it's capable of. So let's say AI writing assistants more capable dialogue agents unsupervised
 translation blah blah blah. They also say a few kind of let's say bad implications.
 Generate misleading news articles impersonate others online automate the production of abusive
 or fake content to post on social media automate the production of spam or phishing content.
 They liken it to a system called deep fakes which generates really well crafted let's
 say videos of people. So that the kind of they frame it in a way as this could be used
 for dangerous things and they say they aren't releasing they're only releasing the small
 version of GPT-2 along with the code. They're not releasing the data set training code or
 the GPT-2 this is the big model the model of weights right. And they do this they cite
 safety concerns. So I mean the community basically is going nuts over this statement this decision
 not release basically the code or the model or the data set to the world. And so if you
 search on Twitter for GPT-2 then everyone basically has an opinion of whether or not
 this is a good thing or not apart from people testing it out. So they've given access to
 like a selected set of journalists to an API where they can query the model. So there are
 some samples flying around but basically people are just debating whether or not this is a
 good thing or not and I mean it's just hilarious to go along with this and to just read all
 the people having opinions. I mean I've given my opinion as well just chime in it's a fun
 ride especially like this post here on reddit machine learning says should I release my
 NIST model or keep it closed source fearing malicious use. Today I trained a 23,000 layer
 ResNet got a 99% accuracy and MNIST. I'd love to share the model but I fear it being used
 maliciously. What if it is used to read documents by the Russians? What are your thoughts? I
 mean yeah this is in essence I mean in essence it's that right. It's like yeah come on. So
 I can just give my opinion up front. I think a lot of things came together here and I think
 that this being OpenAI being kind of this initiative I mean it's never been there before
 they're not an academic institution. They're not a company but still they you know they're
 researchers they want to have a career so there's lots of pressures on them. There's
 pressure to further publish so I think that that's kind of an underlying motivation to
 not release your model and your code and your data set is you actually you know there's
 a lot of work in this and you actually might want to get more than one paper out of it
 so keeping these things to yourself is basically a surefire guarantee you're going to get another
 two, three, four, five papers out of this data or model or whatever. It's also a good
 way to kind of generate press if you basically say oh we're not releasing but we have this
 really good model and there's one thing on Twitter right I mean you can't probably can't
 find it but says like step one my model is state of the art step two my model is state
 of the art but generalizes better to other tasks step three my model does the same thing
 but with fewer parameters and step four my model is so good I can't even talk about it.
 So basically I think a lot of things came together this press generating the pressure
 to create more kind of papers out of it and genuinely security concerns. I think being
 open AI and open AI kind of was established as a way to let's say the demo like their
 statutes pretty clearly say we want to open AI and research it in ethical use and you
 have backers like Elon Musk that talk all the time about kind of safety related issues
 in AI. I think there's a lot of pressure on these people to with everything they do basically
 have an ethical component. So everything they do is kind of scrutiny to this does this have
 ethical implications and where can we basically stand out from the rest of the community by
 doing something it doesn't need to be more ethical just needs to be different with an
 ethical reason behind reasoning behind it and this I think this is it I think there's
 a lot of things coming together I don't I don't think anyone like maliciously thought
 oh this you know we're gonna do this it's gonna generate so much press or I don't think
 anyone actively thought ah we'll just keep it secret we're gonna get so much more papers
 out of it I honestly think that the reasoning was there's a lot of you know a lot of pressure
 to do this ethical things when there's there's not if you think about it it's yeah it's
 a good language model and it can generate text but you can also hire you know people
 to generate text to generate fake news to do phishing and spam it's just a bit more
 efficient now right and yeah so it's it's unprecedented that you don't you don't release
 this research kind of cold war style so it's not really dangerous to release this and it's
 just delaying the inevitable basically but I think that the pressure the pressure was
 on and they made a decision that they thought was in line with their values and I think
 the this just neatly aligns with the underlying the other benefits to them that yeah all right
 so let's dive into the paper the paper is actually not you know too too much content
 there what they basically say so far is that a lot of a lot of these kind of papers on
 these tasks they they say the dominant approach to creating ML systems is to collect a data
 set of training examples demonstrate correct behavior train a system to imitate test its
 performance on in IID samples so they basically say the there's kind of the single task training
 on single domain data sets is a major contributor to the lack of generalization observed in
 current systems so they basically say these language systems they don't generalize like
 a QA system might be trained on a QA task but it you know has nothing to do with the
 task is basically a little bit different and even in multitask learning they say multitask
 learning is a promising framework but also it's kind of say it's still nascent and there's
 only very few different tasks right to do so they basically say you need basically a
 big big unsupervised model and it will implicitly learn all of the kind of special tasks and
 yeah so they say there there are approaches that basically basically learn these language
 models but then still require supervised training so basically fine-tuning this has been the
 this is for example the bird paper we discussed in the in the last video or two two videos
 ago that learns a giant language model but then does fine-tuning for each of these tasks
 and gets really well what they want to do here is basically simply learn a language
 model and then investigate whether or not the language model can perform downstream
 tasks in a zero-shot setting that means without any parameter or architecture modification
 so no fine-tuning all right so what they do so basically what a language model is if for
 those who don't know it's it's if you have a sequence of text let's say a b c d e these
 are words let's act like some actual words the cat sat on the mat and so on and you and
 you a language model is you kind of remove the end of the sentence at some point and
 ask the model what comes next right that's a language model I mean there's different
 kinds of language models specific language ones but that's the basic the basic thing
 so the you just ask the model what's next so you can you can do a lot of unsupervised
 training because you don't need a label data set for this you simply need a text corpus
 and that's basically all they do they use transformers which we've also discussed in
 attention is all you need paper so if you if you don't know what transformers are go
 back and look at that yeah all right so basically they say a lot of these special tasks like
 translation and question answering can be framed in language model way for example if
 you simply input if this is your text translate to French and then the English text and then
 the French text right and then at at test time basically you leave away the French text
 you simply ask the language model what comes next right if and its input is translate to
 French and then English text this is the translation framed as a language model task because you
 can specify the task that the language allows to do also as language so this is quite this
 is quite an interesting approach and one they exploit here and they basically say well since
 in a large and diverse corpus of web pages that they collect here some there is going
 to be some websites that basically all do translation from English to French and the
 model can learn from that so here in this paragraph they basically list examples of
 naturally occurring demonstrations of English to French and French to English translation
 found throughout the training data set so basically this is this is how the model could
 learn let's just look at one I hate the word perfume Bursas it's somewhat better in French
 right so there's a way in just an unsupervised setting where the language model could learn
 right if you just cross out this word at the end and you just ask the model what comes
 next right the model sees I hate the word perfume Bursas it's somewhat better in French
 period colon then the model has to put something there and the most logical continuation is
 to put the French word for perfume right so that that's kind of how they frame translation
 and these other tasks in language model way all right so they talk about the training
 data set which is a major component here they say they make a new training data set because
 all of the current ones aren't sufficient they say most prominent source of diverse
 nearly unlimited text is web scripts such as common crawl while these archives are many
 orders of magnitude larger than current language modeling datasets they have significant data
 quality issues so to say content are mostly unintelligible and so on so they basically
 describe here how they scrape a new web scrape which emphasizes document quality they go
 on reddit basically and scrape all outbound links from reddit that have received at least
 three karma which means that it yeah three up votes for a post of a link which basically
 means that three humans agreed that this was a good link so so they that's that's how they
 collect the data set resulting data set web stack web text contains text subset of the
 45 million links they then kind of clean this and scrape it down and remove some stuff and
 they they end up with a large corpus right and then they talk about how they represent
 the input which is byte pair encoding style it's not exactly by parent coding it's a
 byte pair encoding inspired encoding we won't make a video about this by itself because
 it's really interesting but basically it's you can think of it as tokenization and pre-processing
 right then they say they they show their models so architecture hyperparameters basically
 these are these are their models this is the smallest one this second smallest one they
 say it's the same size as BERT so the the language model by google that we've looked
 at and then the largest one 1.5 billion parameters I mean that's huge and yeah they say it's
 ten times larger than the previous so the first one is their previous model and this
 now is this is the GPT-2 model that that gets all these these nice results so they do experiments
 first they do experiments on language modeling itself right so they train on their on their
 corpus and then they evaluate on a bunch of other language modeling corpus so these up
 here are language modeling corpuses and the state of the art is in this row and then you
 just look at basically the bottom row compare to their largest model this this is perplexity
 where it says PPL and the I think this here is is is accuracy so perplexity lower is better
 which you can you can see here the previous state of the art was 39 on wiki text 2 they
 get to 18 with accuracy obviously higher is better so the the kind of previous accuracy
 in Lombarda was 59 they get to 63 basically improve everything except for this 1 billion
 word corpus and they they also explain why they say this is the most heavily pre-processed
 text and so on so that basically they basically are really good at language modeling even
 though they train on a different data set that's the the point right the point is they
 train on their own corpus and then they go and just evaluate on the test set of these
 of these new of these new tasks and they become better basically than the models that trained
 on the training data set of that particular task all right so they they do a number of
 further experiments where they basically show that the model has now learned kind of implicitly
 learned a number of different tasks so let's look at for example summarization this just
 want to show an example of how you can do this so summarization summarization task is
 you get a long text you need to produce a short text and that short text is then compared
 to short texts that humans wrote when the task was to summarize the long text and you
 get points on how much your text overlaps with these human texts all right so they they
 say we test gpt2's ability to perform summarization on the cnn and daily mail data set to induce
 summarization here's what i found interesting we add the text tldr after the article and
 generate 100 tokens right then they say they need to reduce repetition and so on but basically
 this this this is right this is the way you can frame summarization by text input so i
 find this just kind of a really nice way to think about these problems the fact that instructions
 of the task can be given as text this is a very nice example here so basically you you
 put you as input you put the entire article right and so you here is the the cnn article
 blah blah blah it's super long right and then here you put tldr which is for those who don't
 know it's too long didn't read so people use this this phrase to indicate that then they
 will write a short summary of whatever was before here they will either put this at the
 beginning or at the end of a long text right to to say to people okay if you if you don't
 want to read all this just read this down here um gives you the gist of it which is
 exactly summarization so if you then take this away and ask the language model what's
 here right basically throughout the training corpus the language model will have encountered
 such pieces of text with a tldr in it and the language model might have learned that
 whatever is down here is a short version of whatever is up here and thereby if you then
 ask the language model what comes next here right the language model might learn aha i
 need to summarize whatever is above and that's the my best shot at completing at at answering
 the question what comes next and yeah so they get you know surprisingly good results um
 from from this so they say on the commonly reported rouge 12l metrics the generated summaries
 only begin to approach the performance of classic neural baselines just barely outperforms
 selecting three random sentences from the article uh but but um still it it um while
 qualitatively the generations resemble summaries they often focus on recent content from there
 to color confuse specific details so this is kind of a task where it kind of worked but
 not really um but i just find it it's really interesting that that it it kind of how they
 frame the task and how this can still so it still kind of works and that's the the gist
 here in all of these tasks is also with like translation they obviously don't get near
 the performance of a system specifically trained to do this task but they all also always say
 it kind of works right it's sort of sort of it learns something and their entire point
 of this paper is to say well look um yeah the the the diversity of tasks the model is
 able to perform and i would say kind of perform in a zero shot setting suggests that high
 capacity models trained to maximize the likelihood of a sufficiently varied text corpus begin
 to learn how to perform a surprising amount of tasks without the need for explicit supervision
 so yeah their entire point is if we train on such varied data that kind of um that spans
 the entire range of human language expression the the kind of tasks we want these systems
 to do will be learned implicitly so basically it points to let's get an even bigger corpus
 let's get even bigger models and we might get even better unsupervised zero shot way
 in these kind of special tasks and general language understanding all right so that that
 was basically i've jumped over a lot of points but i encourage you to look into this to look
 into the specific experiments they're really interesting the way how they framed things
 and um give just just shout your opinion around about whether or not the publishing is a good
 thing or not it's really funny i love it um and with that have a good day