[ML News] AI models that write code (Copilot, CodeWhisperer, Pangu-Coder, etc.)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper mlnews ml news copilot codewhisperer copilot legal copilot github google code ai code ai coding ai code assistant what is deep learning
#mlnews #ai #copilot 

OUTLINE:
0:00 - Intro
0:20 - Copilot Now Generally Available
3:20 - FOSS Org leaves GitHub
6:45 - Google's Internal ML Code Completion
9:10 - AI Trains Itself to Code Better
14:30 - Amazon CodeWhisperer in Preview
15:15 - Pangu-Coder: A New Coding Model
17:10 - Useful Things

References:
Copilot Now Generally Available
https://github.blog/2022-06-21-github-copilot-is-generally-available-to-all-developers/

FOSS Org leaves GitHub
https://www.theregister.com/2022/06/30/software_freedom_conservancy_quits_github/
https://sfconservancy.org/blog/2022/jun/30/give-up-github-launch/
https://sfconservancy.org/GiveUpGitHub/
https://sfconservancy.org/docs/SupportGiveUpGitHub-README-snippet.md

Google's Internal ML Code Completion
https://ai.googleblog.com/2022/07/ml-enhanced-code-completion-improves.html

AI Trains Itself to Code Better
https://arxiv.org/abs/2207.14502
https://arxiv.org/pdf/2207.14502.pdf

Amazon CodeWhisperer in Preview
https://aws.amazon.com/blogs/aws/now-in-preview-amazon-codewhisperer-ml-powered-coding-companion/
https://aws.amazon.com/codewhisperer/
https://aws.amazon.com/codewhisperer/features/

Pangu-Coder: A New Coding Model
https://arxiv.org/abs/2207.11280
https://arxiv.org/pdf/2207.11280.pdf

Useful Things
https://github.com/qdrant/quaterion
https://github.com/facebookresearch/torchdim
https://www.mosaicml.com/blog/farewell-oom
https://github.com/hristo-vrigazov/mmap.ninja#when-do-i-use-it

Links:
Homepage: https://ykilcher.com
Merch: https://ykilcher.com/merch
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
LinkedIn: https://www.linkedin.com/in/ykilcher

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 GitHub Copilot is now available to all developers while a big open source community is leaving it behind. But not only GitHub but also Google and Amazon are jumping into the game of AI assisted source code generation. Welcome to ML News. Today we talk all about models that generate source code and that assist developers in writing source code. The GitHub blog released a post last month saying GitHub Copilot is generally available to all developers. Copilot is obviously the product by GitHub based on OpenAI codecs model that suggests source code completions to you based on a large language model that's been trained on all of public GitHub repositories. This is I have to say a really cool product. I was part of the closed beta and it was a game changer, especially if you write any sort of boilerplate code, this thing will just write an entire function for you. It will write your tests, it will write your doc strings, it will write your assertions and your error messages. It's just very, very good for a specific subset of programming. But nevertheless, that subset is making a lot of difference in a lot of people's lives. So the product now is out of this beta and is available to all developers for a price. So it's 10 bucks a month or 100 a year, which I feel is reasonable if you are a programmer by profession, this thing is potentially going to make you a lot more productive than the 10 bucks a month. It is free for verified open source projects and for verified students. Now this is AI news and not necessarily and not always AI shilling. So GitHub has not been without controversy. Currently we have reported on this GitHub has been trained on a lot of code, including open source code, including code that has been licensed under various copy left licenses with the intention that whatever products are made from that code are also free and available to the community. These copy left licenses such as the GPL are specifically made such that no company can just grab that code and then resell it as a product because it's based on the work of a lot of unpaid volunteers. Essentially, copilot is doing exactly that it's taking a lot of code that's publicly accessible yet licensed under such licenses, taking it in training a large language model on it and then selling that to you as a product. Now this is a legal gray area. For example, you as a programmer are perfectly entitled to go look at a piece of code even if it's under the GPL and learn from that piece of code and then implement that same algorithm in your own way in your own code. That is not a violation of copyright is a different story if that algorithm is patented but in terms of copyright and copy left, you're perfectly fine doing that. So it's perfectly reasonable to say that training a large language model on that code that then sort of takes bits and pieces learns from it and then synthesizes its own version from what it learned is a lot like a human doing that same thing. However, obviously it being automated and it being you know, cranked up to 11 in size and speed and it then being sold to all the developers out there might be a different story. And that's why the register writes open source body quits GitHub urges you to do the same. This article is about the software freedom conservancy. This is a nonprofit focused on free and open source software, and they are arguing that GitHub is essentially using your work to build its own proprietary system, namely GitHub co pilot and GitHub itself. Remember, the source code of the GitHub website isn't public. So your work as an open source developer essentially goes into GitHub as a product. And that's exactly what a lot of these open source people don't want. So the software freedom conservancy has released a blog post called give up GitHub, the time has come in which they detail that not only they are leaving GitHub, but they tell you to do the same and they are announcing a plan and support structures from them to support people to get away from GitHub and to move to more open source friendly alternatives. Specifically, obviously, the biggest impact is going to make to move the source code hosting away from GitHub to some other place be that either a cloud hosted provider or a self hosted something. And while I recognize that the idea kind of makes sense, if those things are important to you, it seems like a bit useless and pointless, like just as no license is stopping GitHub from scraping its own repositories. If you put your source code on your website, nothing stopping GitHub from just scraping that. It's the same deal a human is allowed to look at it, learn from it and then reimplement it. So is the language model, at least for now. So it seems like the real path forward here would be a legal one in which there could be a license that explicitly states that no training on this data of any sort is allowed, which essentially might amount to just a patent. But I don't know, I'm not a lawyer. So I don't know what can even be done in these kinds of situations. And the boundaries between humans and language models and code assist and whatnot get extremely murky. So language model is an insanely useful product and GitHub has been a absolutely great place for most of open source in the last many, many years. And obviously, as with a lot of free products, there's got to be a way to make money around that. Now, sure, there are various business models around open source, but I'd rather pay for copilot than seeing an ad every time I want to clone a git repo. So there are a lot of questions in the air right here. What's also interesting is that they give you this snippet that they encourage you to put into your readme if you can't move away from GitHub just now saying we are using GitHub under protest. This project is currently hosted on GitHub, we are deeply concerned about using a proprietary system like GitHub to develop our FSS project. Any use of this project code by GitHub copilot past or present is done without our permission. We do not consent to get up use of this project code in copilot. Yes, about as effective as the if you are not the intended recipient of this message, delete this email right now. It does nothing. I mean, it's obviously there to raise awareness. But still, I don't see how even moving away from GitHub will solve the larger issues around this topic. But let me know what you think in the comments. Be happy to hear your opinions. Google released a blog post called ml enhanced code completion improves developer productivity. This is about an internal study that they have done where they augmented their own code completion engine, which is based on very classical code completion, such as what variable names exist, what functions exist, yada, yada, and they augmented that with ml based code completion such as copilot. So they experimented with various flavors such as single line completion, multi line completion, or simply ranking the outputs of the semantic engine that they already had by using a machine learning model. This all is based on a language model architecture, notably it only has point 5 billion parameters. So as tiny modeling current standards, but they say this is due to latency requirements. So that makes a lot of sense. Google has deployed this internally to their developers and have found a great increase in efficiency of programming compared to a control group. Now while it's really cool that a big company can just run these experiments internally on their people, it must suck to be in the control group. One of these like, this is the latest and greatest tech and you know, your company internally only has access to it and then you're like, bam, you're in a control group. I'm sorry for you control groupers. I hope you get access soon. So this blog post here claims that just under 3% of all new code that's added to the Google code base is code that has been accepted by recommendation from a machine learning engine. There's a 6% reduction in coding iteration duration, there's a 7% reduction in context switches such as moving away from the IDE to go look something up and they have about a 25% acceptance rate, which is how often a suggestion pops up versus how often you accept that suggestion. These numbers look a little bit different for multi line suggestions, but still very encouraging. Now while this is really cool, as I said, it's only available Google internally currently, it also has been trained on their internal code base, which is huge, we're left to see whether or not that or something like this is going to be available to the general public anytime soon. As we saw with copilot, there is definitely money to be made with ML supported code completion, but Google might just be happy with the increase in productivity of their own workforce. And that's going to make them a lot of money by itself. There's a new paper called language models can teach themselves to program better. Now this is a little bit different from code completion as it deals with programming puzzles as specifically programming puzzles that are formulated as tests in programming languages. So the general structure is that the problem is posed as a function f that takes one parameter and checks the validity of that parameter. Somehow, you can specify a lot of things as taking a solution and then verifying it. I mean, I guess you can specify any sort of problem in that way. And then the solution to that would be a function called g right here, g gets access to the source code of f and is then supposed to write code that returns something that's then fed into f that's going to make f true bit more complicated example is down here. So f will accept an x and check if that x is a palindrome. Now there can be more arguments right here, for example, the length of that palindrome and g does get access to these arguments as well. But still the same principle g is going to get access to the source code of f is can analyze it as much as it wants and then has to come up with its own source code that makes f go true. So the problem f here is in fact, the finding of a palindrome with exactly n copies of each of a given list of substring. And so you can see right here that the solution is you simply take n of each you join them and then you add the reverse to it. I guess that wouldn't work if either of the arguments here are themselves a palindrome, because then technically that string would also appear in that part right here. Or if like the cross here like the cross boundary, well, you see it gets arbitrarily complex, but you get the point. These are illustrative examples. So there is a training set, but it only contains 155 puzzles authored by humans. And the trick here is that not only use AI to solve these puzzles, but you actually use it to generate more of them. So we have lots of open source models and closed source models such as codecs that can generate source code that are pre trained on source code. So the paper prompts these models with a bunch of prefixes from the training set. So here you see that's just the problems, not the solutions. And then the models are tasked to come up with more problems. The next step you use the same language models or different ones to actually solve those generated problems and you give them a bit of time so they can explore a bunch of options which you can automatically verify. Now that leaves you with a large set of automatically created but programmatically verified synthetic puzzles, on which you can then fine tune that language model and start from the top so you can use the same language model potentially multiple times to come up with new problems, new solutions to them verify all of that and then retrain these models again. Now as far as I understand the paper only does one cycle of this and already observes a huge boost, especially on the verified examples. So when they make sure that he generated problems and solutions actually, you know, match and work and return true. In that case, there seems to be a big boost if you retrain these language models. So you can see right here, variant of GPT Neo solves only about 7.5% of the test puzzles when just tasked like that. But if you go through all of the steps, it solves 38.2% of all these puzzles. Now there are several issues right here, obviously information theoretically, you can't just punger out information out of nothing. So whatever these models know, you know, you essentially just feed that back to them with the step in between of actually verifying the code. But given that they've been trained on public code, and a lot of that presumably runs, especially if it's kind of filtered for more higher quality training data, then that check shouldn't be too much of a barrier. So it seems like if we just prompted these models better, we could probably get them to solve a lot more of these programming puzzles, since the knowledge seems to be somewhere in there. And also, there's the other issue that these programming puzzles, you know, humans came up with them and so on, they might not be on GitHub themselves. So deduplication is obviously necessary, but deduplication might not be enough as kind of like the solutions to the problems themselves might be in some way somewhere on GitHub, like in the training data of these models. And that way, if you just prompt them in that direction, there might be some effect right there. I don't know, but it is definitely a cool result. And it seems like if we compare these models correctly, prompt them correctly, and then use additional resources, such as these external verification procedure in order to enhance the training data in order to just just make it better, less noisy, more to the point of what we want, that could be a good way forward to get these large models to do what we want. And it might be an alternative to coming up with smart prompts that just kind of work somehow like the let's think about it step by step trick, like it would be nice if we had a more systematic way of getting these models to do what we want. And I think this paper is a step in that direction. Okay, so Amazon joins the ring of ML powered code completion with its code whisperer product. Now much like copilot, this is a model that generates source code and you can subscribe to it, it integrates with your ID and then you can try it out, you can let it complete source code and suggest stuff. Now it's a little bit different in that they not only want to do completion, but they also claim to do security scans in your code. And it's apparently specifically good at interacting with AWS API, they claim it's trained on open source code, but also on Amazon internal code. Now for now, this product is closed, there's a waitlist, you can put your name on there, no guarantee. But it's interesting to see that yet another company is sort of hopping on this ML based code completion thing. There's another new paper out of Huawei called Pangu coder program synthesis with function level language modeling. This is a system based on the Pangu alpha architecture, which is a Chinese large language model and is much like codex fine tuned on code. Now there are a few notable differences. For example, this paper focuses on solving the human eval data set challenge in the end, which is a Python challenge where you get a description of what a function should do. And then you should generate that function, you also get a bunch of unit tests, it is kinda like stuff that we've seen before, but it's also different. The architecture here is nothing special. It is a decoder only language model that is first trained on on just source code in general, and then fine tuned more and more towards this challenge. One interesting thing is that as they progress, they pay attention to the quality of data, which seems to be quite important in these code completion models. So they verify the abstract syntax tree of Python files. And then as an intermediate step before they actually go to the data set, which is remember human descriptions plus the function body that you're supposed to generate, they do take the doc strings of functions that are of appropriate length as an intermediate like as a proxy task. So they view the doc string as the description, and then they generate the function body from that seems pretty straightforward. And obviously, there is lots of suspicions that things like co pilot are training at least in part on similar things. Now they do have a bunch of other improvements and technical nuances over which I don't want to go in here. But all of this results in models that are smaller than other code generation or other coding competition models yet improve upon their performance, which is pretty cool. So if you're interested, check out the paper, I'll link it in the description. And just a few helpful things for this week. Quaternion is a blazing fast framework for fine tuning similarity learning models. So the specific focus here is on fine tuning these models in a very fast and data efficient way with small data, I should say potentially small data, obviously, you can use large data, but it is possible with small data. This is built on top of pytorch lightning. So it's quite accessible and user friendly. Torch Dim is a project out of pytorch. It's in preview, but it introduces named tensors. So named tensors are a concept of first class dimensions in tensors and things like pytorch. Now the idea here is that instead of you having to remember that the first dimension is the batch dimension and then always address with a zero and just keep that in mind is that you address dimensions specifically. So this introduces a dim type, a type four dimension, for example, batch, and then you can simply use that batch dimension in order to index tensors. This isn't a speed up in runtime or anything like this, it just makes code a whole lot more reasonable and a lot less prone to error. The mosaic ml composer library now has automated gradient accumulation. So they claim that composer lets users seamlessly change GPU types and number of GPUs without having to worry about batch size. CUDA out of memory errors are a thing of the past. I'm not going to believe that I'm sorry, even if you solve every single problem that we know of CUDA out of memory errors will stay with us until the eventual downfall of civilization in the year 2089. But apart from that, with the trainer of composer, you can simply tell it to gradient accumulate automatically gradient accumulation is a concept where you don't pass the full mini batch, you only pass part of it, which I guess is then called a mini mini batch. So the full mini batch, if you wanted to run it, you propagate it and computing the gradient would blow your memory because you're training a transformer that's just too big for your GPU at that batch size. So you can propagate just you know, a few samples or even one sample, you can propagate it and then essentially store those gradients and propagate the next thing and then accumulate those gradients in place until you've passed the entire mini batch and only at the end of passing all the individual samples or subparts, you will then do the gradient update step to your weights. This is a known trick. So essentially, your training behaves as if you were to use the large batch size. And we know that large batch sizes are important for some of the current models, especially the large ones. So it behaves like you train with a large batch size, but you can run it on hardware that can only handle a smaller batch size. Now the trade off here is time so you use the amount of forward passes in time that you split your mini batch into, but it's better than not being able to run it at all. And this library does it automatically. And lastly, M map ninja will store your training files as memory map files, which makes training iteration or evaluation any sort of iteration over these files a lot faster. So here the read me says, when do I use it use it whenever you want to store a sequence of non pi arrays of varying shapes that you are going to read from at random positions very often. So the problem here is that if you have a file on disk with a lot of stuff in it, and you want to read at random positions, then very often the operating system makes you scan that file either from the beginning or from some intermediate large chunk barrier, and that can be very cumbersome. So memory mapping is a way of speeding that up. And this library handles it transparently for you. All right, that was already it for this episode of ML news. Let me know what you think about AI models that code and everything else in the world. As always, stay hydrated. Bye bye.
 GitHub Copilot is now available to all developers while a big open source community is leaving
 it behind.
 But not only GitHub but also Google and Amazon are jumping into the game of AI assisted source
 code generation.
 Welcome to ML News.
 Today we talk all about models that generate source code and that assist developers in
 writing source code.
 The GitHub blog released a post last month saying GitHub Copilot is generally available
 to all developers.
 Copilot is obviously the product by GitHub based on OpenAI codecs model that suggests
 source code completions to you based on a large language model that's been trained on
 all of public GitHub repositories.
 This is I have to say a really cool product.
 I was part of the closed beta and it was a game changer, especially if you write any
 sort of boilerplate code, this thing will just write an entire function for you.
 It will write your tests, it will write your doc strings, it will write your assertions
 and your error messages.
 It's just very, very good for a specific subset of programming.
 But nevertheless, that subset is making a lot of difference in a lot of people's lives.
 So the product now is out of this beta and is available to all developers for a price.
 So it's 10 bucks a month or 100 a year, which I feel is reasonable if you are a programmer
 by profession, this thing is potentially going to make you a lot more productive than the
 10 bucks a month.
 It is free for verified open source projects and for verified students.
 Now this is AI news and not necessarily and not always AI shilling.
 So GitHub has not been without controversy.
 Currently we have reported on this GitHub has been trained on a lot of code, including
 open source code, including code that has been licensed under various copy left licenses
 with the intention that whatever products are made from that code are also free and
 available to the community.
 These copy left licenses such as the GPL are specifically made such that no company can
 just grab that code and then resell it as a product because it's based on the work of
 a lot of unpaid volunteers.
 Essentially, copilot is doing exactly that it's taking a lot of code that's publicly
 accessible yet licensed under such licenses, taking it in training a large language model
 on it and then selling that to you as a product.
 Now this is a legal gray area.
 For example, you as a programmer are perfectly entitled to go look at a piece of code even
 if it's under the GPL and learn from that piece of code and then implement that same
 algorithm in your own way in your own code.
 That is not a violation of copyright is a different story if that algorithm is patented
 but in terms of copyright and copy left, you're perfectly fine doing that.
 So it's perfectly reasonable to say that training a large language model on that code that then
 sort of takes bits and pieces learns from it and then synthesizes its own version from
 what it learned is a lot like a human doing that same thing.
 However, obviously it being automated and it being you know, cranked up to 11 in size
 and speed and it then being sold to all the developers out there might be a different
 story.
 And that's why the register writes open source body quits GitHub urges you to do the same.
 This article is about the software freedom conservancy.
 This is a nonprofit focused on free and open source software, and they are arguing that
 GitHub is essentially using your work to build its own proprietary system, namely GitHub
 co pilot and GitHub itself.
 Remember, the source code of the GitHub website isn't public.
 So your work as an open source developer essentially goes into GitHub as a product.
 And that's exactly what a lot of these open source people don't want.
 So the software freedom conservancy has released a blog post called give up GitHub, the time
 has come in which they detail that not only they are leaving GitHub, but they tell you
 to do the same and they are announcing a plan and support structures from them to support
 people to get away from GitHub and to move to more open source friendly alternatives.
 Specifically, obviously, the biggest impact is going to make to move the source code hosting
 away from GitHub to some other place be that either a cloud hosted provider or a self hosted
 something.
 And while I recognize that the idea kind of makes sense, if those things are important
 to you, it seems like a bit useless and pointless, like just as no license is stopping GitHub
 from scraping its own repositories.
 If you put your source code on your website, nothing stopping GitHub from just scraping
 that.
 It's the same deal a human is allowed to look at it, learn from it and then reimplement
 it.
 So is the language model, at least for now.
 So it seems like the real path forward here would be a legal one in which there could
 be a license that explicitly states that no training on this data of any sort is allowed,
 which essentially might amount to just a patent.
 But I don't know, I'm not a lawyer.
 So I don't know what can even be done in these kinds of situations.
 And the boundaries between humans and language models and code assist and whatnot get extremely
 murky.
 So language model is an insanely useful product and GitHub has been a absolutely great place
 for most of open source in the last many, many years.
 And obviously, as with a lot of free products, there's got to be a way to make money around
 that.
 Now, sure, there are various business models around open source, but I'd rather pay for
 copilot than seeing an ad every time I want to clone a git repo.
 So there are a lot of questions in the air right here.
 What's also interesting is that they give you this snippet that they encourage you to
 put into your readme if you can't move away from GitHub just now saying we are using GitHub
 under protest.
 This project is currently hosted on GitHub, we are deeply concerned about using a proprietary
 system like GitHub to develop our FSS project.
 Any use of this project code by GitHub copilot past or present is done without our permission.
 We do not consent to get up use of this project code in copilot.
 Yes, about as effective as the if you are not the intended recipient of this message,
 delete this email right now.
 It does nothing.
 I mean, it's obviously there to raise awareness.
 But still, I don't see how even moving away from GitHub will solve the larger issues around
 this topic.
 But let me know what you think in the comments.
 Be happy to hear your opinions.
 Google released a blog post called ml enhanced code completion improves developer productivity.
 This is about an internal study that they have done where they augmented their own code
 completion engine, which is based on very classical code completion, such as what variable
 names exist, what functions exist, yada, yada, and they augmented that with ml based code
 completion such as copilot.
 So they experimented with various flavors such as single line completion, multi line
 completion, or simply ranking the outputs of the semantic engine that they already had
 by using a machine learning model.
 This all is based on a language model architecture, notably it only has point 5 billion parameters.
 So as tiny modeling current standards, but they say this is due to latency requirements.
 So that makes a lot of sense.
 Google has deployed this internally to their developers and have found a great increase
 in efficiency of programming compared to a control group.
 Now while it's really cool that a big company can just run these experiments internally
 on their people, it must suck to be in the control group.
 One of these like, this is the latest and greatest tech and you know, your company internally
 only has access to it and then you're like, bam, you're in a control group.
 I'm sorry for you control groupers.
 I hope you get access soon.
 So this blog post here claims that just under 3% of all new code that's added to the Google
 code base is code that has been accepted by recommendation from a machine learning engine.
 There's a 6% reduction in coding iteration duration, there's a 7% reduction in context
 switches such as moving away from the IDE to go look something up and they have about
 a 25% acceptance rate, which is how often a suggestion pops up versus how often you
 accept that suggestion.
 These numbers look a little bit different for multi line suggestions, but still very
 encouraging.
 Now while this is really cool, as I said, it's only available Google internally currently,
 it also has been trained on their internal code base, which is huge, we're left to see
 whether or not that or something like this is going to be available to the general public
 anytime soon.
 As we saw with copilot, there is definitely money to be made with ML supported code completion,
 but Google might just be happy with the increase in productivity of their own workforce.
 And that's going to make them a lot of money by itself.
 There's a new paper called language models can teach themselves to program better.
 Now this is a little bit different from code completion as it deals with programming puzzles
 as specifically programming puzzles that are formulated as tests in programming languages.
 So the general structure is that the problem is posed as a function f that takes one parameter
 and checks the validity of that parameter.
 Somehow, you can specify a lot of things as taking a solution and then verifying it.
 I mean, I guess you can specify any sort of problem in that way.
 And then the solution to that would be a function called g right here, g gets access to the
 source code of f and is then supposed to write code that returns something that's then fed
 into f that's going to make f true bit more complicated example is down here.
 So f will accept an x and check if that x is a palindrome.
 Now there can be more arguments right here, for example, the length of that palindrome
 and g does get access to these arguments as well.
 But still the same principle g is going to get access to the source code of f is can
 analyze it as much as it wants and then has to come up with its own source code that makes
 f go true.
 So the problem f here is in fact, the finding of a palindrome with exactly n copies of each
 of a given list of substring.
 And so you can see right here that the solution is you simply take n of each you join them
 and then you add the reverse to it.
 I guess that wouldn't work if either of the arguments here are themselves a palindrome,
 because then technically that string would also appear in that part right here.
 Or if like the cross here like the cross boundary, well, you see it gets arbitrarily complex,
 but you get the point.
 These are illustrative examples.
 So there is a training set, but it only contains 155 puzzles authored by humans.
 And the trick here is that not only use AI to solve these puzzles, but you actually use
 it to generate more of them.
 So we have lots of open source models and closed source models such as codecs that can
 generate source code that are pre trained on source code.
 So the paper prompts these models with a bunch of prefixes from the training set.
 So here you see that's just the problems, not the solutions.
 And then the models are tasked to come up with more problems.
 The next step you use the same language models or different ones to actually solve those
 generated problems and you give them a bit of time so they can explore a bunch of options
 which you can automatically verify.
 Now that leaves you with a large set of automatically created but programmatically verified synthetic
 puzzles, on which you can then fine tune that language model and start from the top so you
 can use the same language model potentially multiple times to come up with new problems,
 new solutions to them verify all of that and then retrain these models again.
 Now as far as I understand the paper only does one cycle of this and already observes
 a huge boost, especially on the verified examples.
 So when they make sure that he generated problems and solutions actually, you know, match and
 work and return true.
 In that case, there seems to be a big boost if you retrain these language models.
 So you can see right here, variant of GPT Neo solves only about 7.5% of the test puzzles
 when just tasked like that.
 But if you go through all of the steps, it solves 38.2% of all these puzzles.
 Now there are several issues right here, obviously information theoretically, you can't just
 punger out information out of nothing. So whatever these models know, you know, you
 essentially just feed that back to them with the step in between of actually verifying
 the code.
 But given that they've been trained on public code, and a lot of that presumably runs, especially
 if it's kind of filtered for more higher quality training data, then that check shouldn't be
 too much of a barrier.
 So it seems like if we just prompted these models better, we could probably get them
 to solve a lot more of these programming puzzles, since the knowledge seems to be somewhere
 in there.
 And also, there's the other issue that these programming puzzles, you know, humans came
 up with them and so on, they might not be on GitHub themselves.
 So deduplication is obviously necessary, but deduplication might not be enough as kind
 of like the solutions to the problems themselves might be in some way somewhere on GitHub,
 like in the training data of these models.
 And that way, if you just prompt them in that direction, there might be some effect right
 there. I don't know, but it is definitely a cool result. And it seems like if we compare
 these models correctly, prompt them correctly, and then use additional resources, such as
 these external verification procedure in order to enhance the training data in order to just
 just make it better, less noisy, more to the point of what we want, that could be a good
 way forward to get these large models to do what we want.
 And it might be an alternative to coming up with smart prompts that just kind of work
 somehow like the let's think about it step by step trick, like it would be nice if we
 had a more systematic way of getting these models to do what we want. And I think this
 paper is a step in that direction.
 Okay, so Amazon joins the ring of ML powered code completion with its code whisperer product.
 Now much like copilot, this is a model that generates source code and you can subscribe
 to it, it integrates with your ID and then you can try it out, you can let it complete
 source code and suggest stuff. Now it's a little bit different in that they not only
 want to do completion, but they also claim to do security scans in your code. And it's
 apparently specifically good at interacting with AWS API, they claim it's trained on open
 source code, but also on Amazon internal code.
 Now for now, this product is closed, there's a waitlist, you can put your name on there,
 no guarantee. But it's interesting to see that yet another company is sort of hopping
 on this ML based code completion thing. There's another new paper out of Huawei called Pangu
 coder program synthesis with function level language modeling. This is a system based
 on the Pangu alpha architecture, which is a Chinese large language model and is much
 like codex fine tuned on code. Now there are a few notable differences. For example, this
 paper focuses on solving the human eval data set challenge in the end, which is a Python
 challenge where you get a description of what a function should do. And then you should
 generate that function, you also get a bunch of unit tests, it is kinda like stuff that
 we've seen before, but it's also different. The architecture here is nothing special.
 It is a decoder only language model that is first trained on on just source code in general,
 and then fine tuned more and more towards this challenge. One interesting thing is that
 as they progress, they pay attention to the quality of data, which seems to be quite important
 in these code completion models. So they verify the abstract syntax tree of Python files.
 And then as an intermediate step before they actually go to the data set, which is remember
 human descriptions plus the function body that you're supposed to generate, they do
 take the doc strings of functions that are of appropriate length as an intermediate like
 as a proxy task. So they view the doc string as the description, and then they generate
 the function body from that seems pretty straightforward. And obviously, there is lots of suspicions
 that things like co pilot are training at least in part on similar things. Now they
 do have a bunch of other improvements and technical nuances over which I don't want
 to go in here. But all of this results in models that are smaller than other code generation
 or other coding competition models yet improve upon their performance, which is pretty cool.
 So if you're interested, check out the paper, I'll link it in the description. And just
 a few helpful things for this week. Quaternion is a blazing fast framework for fine tuning
 similarity learning models. So the specific focus here is on fine tuning these models
 in a very fast and data efficient way with small data, I should say potentially small
 data, obviously, you can use large data, but it is possible with small data. This is built
 on top of pytorch lightning. So it's quite accessible and user friendly. Torch Dim is
 a project out of pytorch. It's in preview, but it introduces named tensors. So named
 tensors are a concept of first class dimensions in tensors and things like pytorch. Now the
 idea here is that instead of you having to remember that the first dimension is the batch
 dimension and then always address with a zero and just keep that in mind is that you address
 dimensions specifically. So this introduces a dim type, a type four dimension, for example,
 batch, and then you can simply use that batch dimension in order to index tensors. This
 isn't a speed up in runtime or anything like this, it just makes code a whole lot more
 reasonable and a lot less prone to error. The mosaic ml composer library now has automated
 gradient accumulation. So they claim that composer lets users seamlessly change GPU
 types and number of GPUs without having to worry about batch size. CUDA out of memory
 errors are a thing of the past. I'm not going to believe that I'm sorry, even if you solve
 every single problem that we know of CUDA out of memory errors will stay with us until
 the eventual downfall of civilization in the year 2089. But apart from that, with the trainer
 of composer, you can simply tell it to gradient accumulate automatically gradient accumulation
 is a concept where you don't pass the full mini batch, you only pass part of it, which
 I guess is then called a mini mini batch. So the full mini batch, if you wanted to run
 it, you propagate it and computing the gradient would blow your memory because you're training
 a transformer that's just too big for your GPU at that batch size. So you can propagate
 just you know, a few samples or even one sample, you can propagate it and then essentially
 store those gradients and propagate the next thing and then accumulate those gradients
 in place until you've passed the entire mini batch and only at the end of passing all the
 individual samples or subparts, you will then do the gradient update step to your weights.
 This is a known trick. So essentially, your training behaves as if you were to use the
 large batch size. And we know that large batch sizes are important for some of the current
 models, especially the large ones. So it behaves like you train with a large batch size, but
 you can run it on hardware that can only handle a smaller batch size. Now the trade off here
 is time so you use the amount of forward passes in time that you split your mini batch into,
 but it's better than not being able to run it at all. And this library does it automatically.
 And lastly, M map ninja will store your training files as memory map files, which makes training
 iteration or evaluation any sort of iteration over these files a lot faster. So here the
 read me says, when do I use it use it whenever you want to store a sequence of non pi arrays
 of varying shapes that you are going to read from at random positions very often. So the
 problem here is that if you have a file on disk with a lot of stuff in it, and you want
 to read at random positions, then very often the operating system makes you scan that file
 either from the beginning or from some intermediate large chunk barrier, and that can be very
 cumbersome. So memory mapping is a way of speeding that up. And this library handles it transparently
 for you. All right, that was already it for this episode of ML news. Let me know what
 you think about AI models that code and everything else in the world. As always, stay hydrated.
 Bye bye.