[News] OpenAI Model Generates Python Code
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper microsoft openai msbuild build code gpt2 language model completion intellisense intellicode vscode github python code completion smart generate function body docstring name arguments programmer stackoverflow dataset interpolate
This code completion engine can write an entire function from just the name! OpenAI demonstrates what happens when you learn a language model on thousands of GitHub Python repositories.

Source Clip: https://youtu.be/fZSFNUT6iY8
Full Video: https://www.pscp.tv/Microsoft/1OyKAYWPRrWKb
Kite: https://kite.com/
TabNine: https://www.tabnine.com/

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there. So I saw this and probably many of you have seen this. OpenAI was demonstrating at MSBuild basically a GPT-2 language model but trained not on language but on code, on Python code, open source code from GitHub. And so the idea is that the model learns to produce code. And we'll just have a short look at the clip they have here. I'll link the entire clip down. So this is what the human types. Def is palindrome, so the function name, the argument, and the doc string. And now the model is asked to produce the rest of the function and check out. So it's pretty good, right? This is actually to check whether, this is a basic check, whether a string is a palindrome, as long as you can ensure that s is a string and so on. So the model learned this. You can still say maybe that's just interpolating from, you know, something like this is surely in a GitHub repo somewhere. So they go further and they try to say, okay, please give me a function where the palindromes, so return a list indices for elements that are palindromes and at least seven characters. And I personally have searched for this function on GitHub and it does not exist. So what does the model produce? Pretty cool. So this is first of all a list comprehension in Python, which is reasonably complicated, right? And you can see there is this length filter that is greater or equal to seven. And it refers actually back to the is palindrome function that it wrote before. That's pretty cool. Now, this is not like a language model producing, as far as I understand, producing this basically letter by letter or word by word. This goes over the constraints of abstract syntax trees. So it is, I think that's what's happening. They don't have a paper to go along, though I will look into more papers of that sort. They do kind of constrain the model to actually produce valid code. But of course, which variables go where and so on that that's that is completely up to the model. And you see here it understands completely what the user wants. Now, of course, these examples might be cherry picked, right? But it's even for cherry picked examples, still pretty impressive. As I said, I could not find this particular function. So they post two classes here data classes, item and order. And now the model is asked to compute the total order price, which is a method of the of the of the order class. And they stop here. So this is what the human enters. The human enters that just the name of the function, not even the doc string. And the model comes up with the following, compute the total price of the order, including the palindrome. So it does all of that by itself, including the doc string, just from the method. Now you can see pretty much what's happening here. It's kind of like the GPT-2 language model. So what it does is probably it from the method name, it derives this doc string compute total price, right? That's a lot of programmers do this of the order. Order is, of course, the name of the class of self. And here it says including the palindrome discount. And that is probably somewhat pattern match to other functions that have some sort of discount or something like this or one argument that is a number. But the fact that it is also able to see, it adds up the total price per item. And it basically discounts every item. Now it cannot work out that palindrome discount should mean that every item should be a palindrome. That's the only thing it can't work out. It just applies a discount to every single item. Now they go ahead and kind of change that and write the doc strings themselves such that it is clear that apply the discount to items whose name are palindromes. Now the model is again asked to complete this and absolutely crazy. If it's a palindrome, then multiply it with the discount. And if it's not a palindrome, then just add the price. And this final touch here, that's one minus the palindrome discount is added by the programmer. So you can see that this goes towards kind of a symbiosis of human and machine in this way. I don't think the AI will replace programmers, but it certainly is going to be very helpful to automate some of the things or give you suggestions for things that you have to do over and over again. Now I think a lot of these rely on the fact that a lot of programming is redundant still. A lot of people name the function and then in the doc string, they basically repeat the function because they've already intelligently named the function. So technically there doesn't need to be a doc string, but then whatever your style guide comes in, it says there needs to be a doc string. Every argument must be described. Every argument must have a description and a help string and a type, even though it is completely obvious from the names what they do. So if it is completely obvious, I would argue you don't need a doc string. And this is kind of additional information that this model is able to actually sort of make use of. So the fact that a lot of these functions, you can already, the doc string is sort of already the implementation of the function almost. So the distance there is not, it's not like you can say whatever you want. And yeah, so you can see here when it's asked to print the receipt, it just works out. So it's printing, it's doing format strings and whatnot. So it's just learned to do that. I would argue, again, this works. You couldn't just put anything like doc string language is a very specific type of language that programmers use where they basically already sort of implement the method in the doc string. And then the body of the method is just the then really specific code. But as of that, yeah, there's a lot of information already in the doc string in the naming of the function. And it's still pretty impressive, right? So yeah, I just wanted to show you that this already is available, even though not in as big of a form, it can't write giant functions for you, you can't write function bodies, but there are some machine learning based completions already available. So kite is one of them. And tab nine is the other one that I use for now. Both are close sources, I understand it. So that's a bit of a downer there. But these are exactly kind of GPT language models learned on a lot of code. So they can kind of guess what you want and interpolate with your variables in there and so on. I also found this when I searched this comparison here kite versus tab nine. And you see it starts off Yeah, but these are I think these are kind of auto generated. So when you look at the video, you get tab nine is correct. But that kite, it's an actual video review of a kite. Yeah, so, you know, who knows. But what I wanted to do is basically show you a bit of the power of tab nine. Let me get this out of the way. So a while back ago, I live coded this session right here where I where I implemented a sentiment classifier from scratch using hugging face libraries. And I thought we would just play around in here a bit to see what the tab nine could do for us. Alright, so I have tab nine in here together with a bunch of other stuff, I have to admit, so I'm not sure how this is going to turn out. So let's say we wanted to compute the loss here. Let's say we wanted to compute square loss, square loss, you see that tab nine immediately kind of turns up. I've not tried this, I'm impressed. So it says it estimates loss here. And no, that's maybe not what I want. So I'll go with square. Now this is a language server suggestion. So and you can see right here, even though I don't have these variables, kind of tab nine will suggest train loss and validation loss. So let me start a new file right here. Just to see what we can get this thing to do without doing anything. So and so let's say we'll import OS and we'll say if name. So tab nine auto suggests that. And it auto suggests that we should write main here, right. And it knows that a lot of people then call a function called main. So we should probably do that def main, right. Sorry. Okay, let's go with the following, we'll say, we'll try the same thing they did, right. So we'll say we have a data class order, it has, let's say price float and a name string. And order one is an order with the price of five and the name of hello. And we can print CC tab nine, tab nine, if you can see that, it's closely suggested to print order dot price order one dot price. So it can it can see that we kind of want that. How do I select that? Right here. See, order two equals so we'll get another order right here. Seven. Hello. Let's get it with order two. Let's do that. Orders equals order one, order two. So total price, total price equals zero for order in. Wow, did you see that? Price, total price equals zero for order in. Wow, did you see that? In, I can't get it anymore. In orders, that's what tab nine says. Total price plus equals order dot price. So it is already pretty smart, I would argue. Print. And there you saw that total price was suggested. How can I, I don't know how to select this. But I'll figure it out. I'm not that advanced yet. So you can see this already sort of works. And I think it's pretty cool already. And I'm very excited to see what kind of how far people can push this because I think this code generation kind of inferring what you want is only at the beginning right now. And it's for sure going to be a very, very, very, very, very, very, very interesting, very, very, very interesting thing to come more. And yeah, with that, bye bye.
 Hi there. So I saw this and probably many of you have seen this. OpenAI was demonstrating at MSBuild
 basically a GPT-2 language model but trained not on language but on code, on Python code,
 open source code from GitHub. And so the idea is that the model learns to produce code. And
 we'll just have a short look at the clip they have here. I'll link the entire clip down. So this is
 what the human types. Def is palindrome, so the function name, the argument, and the doc string.
 And now the model is asked to produce the rest of the function and check out.
 So it's pretty good, right? This is actually to check whether, this is a basic check, whether a
 string is a palindrome, as long as you can ensure that s is a string and so on. So the model learned
 this. You can still say maybe that's just interpolating from, you know, something like
 this is surely in a GitHub repo somewhere. So they go further and they try to say, okay, please give
 me a function where the palindromes, so return a list indices for elements that are palindromes and
 at least seven characters. And I personally have searched for this function on GitHub and it does
 not exist. So what does the model produce? Pretty cool. So this is first of all a list comprehension
 in Python, which is reasonably complicated, right? And you can see there is this length filter
 that is greater or equal to seven. And it refers actually back to the is palindrome function that
 it wrote before. That's pretty cool. Now, this is not like a language model producing, as far as I
 understand, producing this basically letter by letter or word by word. This goes over the
 constraints of abstract syntax trees. So it is, I think that's what's happening. They don't have a
 paper to go along, though I will look into more papers of that sort. They do kind of constrain
 the model to actually produce valid code. But of course, which variables go where and so on that
 that's that is completely up to the model. And you see here it understands completely what the user
 wants. Now, of course, these examples might be cherry picked, right? But it's even for cherry
 picked examples, still pretty impressive. As I said, I could not find this particular function.
 So they post two classes here data classes, item and order. And now the model is asked to compute
 the total order price, which is a method of the of the of the order class. And they stop here. So
 this is what the human enters. The human enters that just the name of the function, not even the
 doc string. And the model comes up with the following, compute the total price of the order,
 including the palindrome. So it does all of that by itself, including the doc string, just from
 the method. Now you can see pretty much what's happening here. It's kind of like the GPT-2
 language model. So what it does is probably it from the method name, it derives this doc string
 compute total price, right? That's a lot of programmers do this of the order. Order is,
 of course, the name of the class of self. And here it says including the palindrome discount.
 And that is probably somewhat pattern match to other functions that have some sort of discount
 or something like this or one argument that is a number. But the fact that it is also able to see,
 it adds up the total price per item. And it basically discounts every item. Now it cannot
 work out that palindrome discount should mean that every item should be a palindrome. That's the only
 thing it can't work out. It just applies a discount to every single item. Now they go ahead and kind
 of change that and write the doc strings themselves such that it is clear that apply the discount to
 items whose name are palindromes. Now the model is again asked to complete this and absolutely crazy.
 If it's a palindrome, then multiply it with the discount. And if it's not a palindrome,
 then just add the price. And this final touch here, that's one minus the palindrome discount
 is added by the programmer. So you can see that this goes towards kind of a symbiosis of human and
 machine in this way. I don't think the AI will replace programmers, but it certainly is going
 to be very helpful to automate some of the things or give you suggestions for things that you have
 to do over and over again. Now I think a lot of these rely on the fact that a lot of programming
 is redundant still. A lot of people name the function and then in the doc string, they basically
 repeat the function because they've already intelligently named the function. So technically
 there doesn't need to be a doc string, but then whatever your style guide comes in, it says there
 needs to be a doc string. Every argument must be described. Every argument must have a description
 and a help string and a type, even though it is completely obvious from the names what they do.
 So if it is completely obvious, I would argue you don't need a doc string. And this is kind
 of additional information that this model is able to actually sort of make use of. So the fact that
 a lot of these functions, you can already, the doc string is sort of already the implementation
 of the function almost. So the distance there is not, it's not like you can say whatever you want.
 And yeah, so you can see here when it's asked to print the receipt, it just works out. So it's
 printing, it's doing format strings and whatnot. So it's just learned to do that. I would argue,
 again, this works. You couldn't just put anything like doc string language is a very specific type
 of language that programmers use where they basically already sort of implement the method
 in the doc string. And then the body of the method is just the then really specific code.
 But as of that, yeah, there's a lot of information already in the doc string in the naming of the
 function. And it's still pretty impressive, right? So yeah, I just wanted to show you that this
 already is available, even though not in as big of a form, it can't write giant functions for you,
 you can't write function bodies, but there are some machine learning based completions
 already available. So kite is one of them. And tab nine is the other one that I use for now.
 Both are close sources, I understand it. So that's a bit of a downer there. But these are
 exactly kind of GPT language models learned on a lot of code. So they can kind of guess what you
 want and interpolate with your variables in there and so on. I also found this when I searched this
 comparison here kite versus tab nine. And you see it starts off Yeah, but these are I think these
 are kind of auto generated. So when you look at the video, you get tab nine is correct. But that
 kite, it's an actual video review of a kite. Yeah, so, you know, who knows. But what I wanted to do
 is basically show you a bit of the power of tab nine. Let me get this out of the way. So
 a while back ago, I live coded this session right here where I where I implemented a sentiment
 classifier from scratch using hugging face libraries. And I thought we would just play around in here a
 bit to see what the tab nine could do for us. Alright, so I have tab nine in here together with
 a bunch of other stuff, I have to admit, so I'm not sure how this is going to turn out. So let's say
 we wanted to compute the loss here. Let's say we wanted to compute square loss, square loss, you see
 that tab nine immediately kind of turns up. I've not tried this, I'm impressed. So it says it
 estimates loss here. And no, that's maybe not what I want. So I'll go with square. Now this is a
 language server suggestion. So and you can see right here, even though I don't have these
 variables, kind of tab nine will suggest train loss and validation loss. So let me start a new
 file right here. Just to see what we can get this thing to do without doing anything. So and so
 let's say we'll import OS and we'll say if name. So tab nine auto suggests that. And it auto
 suggests that we should write main here, right. And it knows that a lot of people then call a
 function called main. So we should probably do that def main, right. Sorry. Okay, let's go with
 the following, we'll say, we'll try the same thing they did, right. So we'll say we have a data class
 order, it has, let's say price float and a name string. And order one is an order with the price
 of five and the name of hello. And we can print CC tab nine, tab nine, if you can see that,
 it's closely suggested to print order dot price order one dot price. So it can it can see that
 we kind of want that. How do I select that? Right here. See, order two equals so we'll get another
 order right here. Seven. Hello. Let's get it with order two. Let's do that. Orders equals order one,
 order two. So total price, total price equals zero for order in. Wow, did you see that?
 Price, total price equals zero for order in. Wow, did you see that?
 In, I can't get it anymore. In orders, that's what tab nine says. Total price plus equals
 order dot price. So it is already pretty smart, I would argue.
 Print. And there you saw that total price was suggested. How can I, I don't know how to select
 this. But I'll figure it out. I'm not that advanced yet. So you can see this already sort of works.
 And I think it's pretty cool already. And I'm very excited to see what kind of how far people
 can push this because I think this code generation kind of inferring what you want is only at the
 beginning right now. And it's for sure going to be a very, very, very, very, very, very, very
 interesting, very, very, very interesting thing to come more. And yeah, with that, bye bye.