AlphaCode - with the authors!
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper alphacode alpha code deepmind deepmind code deepmind alphacode alphacoder codex copilot ai code ai programmer ai competitive programming ai leetcode machine learning leetcode deepmind leetcode codeforces large scale sampling language models language models for code ai python programmer deep mind fuzzing google deepmind competitive programming ai interview
#ai #alphacode #deepmind

An interview with the creators of AlphaCode!
Paper review video here: https://youtu.be/s9UAOmyah1A

OUTLINE:
0:00 - Intro
1:10 - Media Reception
5:10 - How did the project go from start to finish?
9:15 - Does the model understand its own code?
14:45 - Are there plans to reduce the number of samples?
16:15 - Could one do smarter filtering of samples?
18:55 - How crucial are the public test cases?
21:55 - Could we imagine an adversarial method?
24:45 - How are coding problems even made?
27:40 - Does AlphaCode evaluate a solution's asymptotic complexity?
33:15 - Are our sampling procedures inappropriate for diversity?
36:30 - Are all generated solutions as instructive as the example?
41:30 - How are synthetic examples created during training?
42:30 - What were high and low points during this research?
45:25 - What was the most valid criticism after publication?
47:40 - What are applications in the real world?
51:00 - Where do we go from here?

Paper: https://storage.googleapis.com/deepmind-media/AlphaCode/competition_level_code_generation_with_alphacode.pdf
Code: https://github.com/deepmind/code_contests

Abstract: Programming is a powerful and ubiquitous problem-solving tool. Developing systems that can assist programmers or even generate programs independently could make programming more productive and accessible, yet so far incorporating innovations in AI has proven challenging. Recent large-scale language models have demonstrated an impressive ability to generate code, and are now able to complete simple programming tasks. However, these models still perform poorly when evaluated on more complex, unseen problems that require problem-solving skills beyond simply translating instructions into code. For example, competitive programming problems which require an understanding of algorithms and complex natural language remain extremely challenging. To address this gap, we introduce AlphaCode, a system for code generation that can create novel solutions to these problems that require deeper reasoning. Evaluated on recent programming competitions on the Codeforces platform, AlphaCode achieved on average a ranking of top 54.3% in programming competitions with more than 5,000 participants. We found that three key components were critical to achieve good and reliable performance: (1) an extensive and clean competitive programming dataset for training and evaluation, (2) large and efficient-to-sample transformer-based architectures, and (3) large-scale model sampling to explore the search space, followed by filtering based on program behavior to a small set of submissions.

Authors: Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu and Oriol Vinyals

Links:
Merch: http://store.ykilcher.com
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hey, this is an interview with the authors of the Alpha Code paper by DeepMind. This is a crazy system. It does automated competitive programming and is about as good as an average human in real competitions, which is crazy. In case you haven't seen it, I've made a comprehensive paper review of this paper in the last video. So be sure to check that out because the authors that I'm interviewing today have also seen that video and were able to dive right into the matter answering any questions, any criticisms and so on. You're also able to get a behind the scenes look into what things went wrong during this research, things that didn't work out, things that were red herrings and much more. We also talk about how the project came to be and how the authors dealt with the immense media reaction that followed the release. Let me know how you like these types of videos. Having the authors on is a huge privilege and I'm absolutely sure you'll learn something useful from this conversation. If you like content like this, don't forget to leave a like, subscribe, tell me what you think in the comments and I'll see you around. Bye bye. Yeah, hi everyone. Welcome back. I'm here today with Rémy LeBlanc and Peter Choi, who are authors of the competition level code generation with Alpha Code paper. I'm just going to call it the Alpha Code paper. Everyone's excited about this paper. So much hype around it and it's very cool to have the authors with me. So Rémy and Peter, thank you very much for being here. Thanks for having us. Thanks a lot for having us. Yeah, we're quite happy to be doing this with you today. So the paper, obviously, given that the machine learning community and the programmer community intersect in large parts and then the competitive programming scene also is kind of known for not being the most humble. Obviously, let's say, there was quite a bit of hype, quite a bit of media reception around the paper. Did you expect anything like this and how did you experience sort of how the paper was received in public? I guess I can take that one for a start, Peter. So I think overall, we've been fairly happy with how the paper has been received, right? People have been talking a lot about the ideas that we put forward and the results that what we think is fairly impressive for what we're trying to do is nowhere near what might have been reported in some news outlets. So we did expect that there was going to be positive reactions, negative reactions and a bit of misunderstandings, probably. But I think overall, we've been fairly happy. Yeah, I think we spent a few hours, maybe even a day or two after we released the paper, just kind of watching with popcorn what was going on. And yeah, that was pretty enjoyable. But yeah, overall, I'd say I'm pretty pleased. Do you want to maybe just as an opportunity to... Did you hear like crass overstatements you said, you know, some people said a bit more than what you actually did. So is there something that you saw that was like really where you say, no, this is actually, this is wrong. It's too much, you know, rather than just selling it very prettily. Anything you sort of want to bring down to earth. I think I can definitely add one thing there. I think the biggest thing that I noticed and like quite a common mistake was to like overstate our result as DeepMind, you know, has an algorithm which is as good as an average programmer. But like really, the right answer is, it's average competitive. You know, we get the same results as an average competitive programmer. And those are like huge, huge, there's a huge difference there. But you know, that distinction can be like a bit nebulous if you're not familiar with the programming or competitive programming. So that's the one, the main thing I think which becomes the top of my list. Yes, of course, like most of the most of your job as a software programmer isn't actually writing code, right? It's reading code, understanding code, thinking about how to achieve whatever it is you want to achieve, right? So we focus on a much, much narrower scope in this paper where we have a very precise description of what we want to do. We have examples, we have constraints, etc. Which to us is a very interesting proxy for problem solving. But it's very far from the full job of an actual developer. Yeah, I was, I mean, I was, I think even with the with the correcting the record, it is still very impressive. And I think before we before the recording, we talked about that also you seem to have been a bit surprised at how far you were able to get with this system. Could you tell us a little bit about the just the process of, you know, how did you start out? What did you do? For example, codecs or copilot from GitHub. And I have to say it's like is really good. Like it's, I think it's it's a game changer if the UI is cleaned up a little bit and models like this will be, you know, I think assisting programmers a lot. But how did you go from like that? Were you even aware of codecs copilot? And how did you get to to alpha code? And what did you expect? Right, so I think and I mean, I wasn't there from the very beginning of the of the problem. But I think we've always been focusing on a slightly different approach than what codecs and copilot are doing. I think we're really interested in this aspect of problem solving and we were really interested in this aspect of generalization. We wanted to solve unseen problems and come up with novel solutions to things that the model hadn't seen during training. And so competitive programming was sort of a natural target for us. And then we started getting a bit of traction and we set ourselves what we thought to be almost an impossible goal. But we thought we needed to be ambitious to really, really push ourselves and push the push the methods. And so our level of confidence in whether or not we're going to achieve this fluctuated during the course of the project. At some points we had high points and we had low points. Some points we're convinced we're going to succeed. At some points we had pretty severe doubts. But yeah, in the end, we managed to get all the way across the finish line. I think one thing I'd add to that is I think this is the first project where I worked on which had quite a strict adherence to looking at a particular metric quite regularly. And I think that really helped us incorporate ideas that were happening, that were being researched within DeepMind and outside of DeepMind. So I think that was really worthwhile and something that we've learned to value quite a lot in working on these ambitious projects. It's cool if you have some sort of a North Star, right? At least you know where you want to get. I think with most projects it's even ill-defined kind of where the end goal is. And I think it's probably half the game in academia and also projects as such. So I've made this little overview and intro to your paper. Did you feel that was accurate? Is there anything missing? You want to amend on how the system works? Any wrong emphasis that I've set? I don't think there's anything wrong with what you described. And I was fairly impressed that you managed to sort of distill this massive paper down to a reasonable size in terms of the video. So yeah, I think I was quite happy with the way you described it. Of course, opportunities to get into more details by reading the paper itself, especially on the maybe on the method section. But overall, it was really good. I was really impressed as always. Yeah, I generally love your videos, Yannick. So it's a really easy way to get an overview of a paper and decide if you want to read it yourself at all. And yeah, this was kind of not an exception. Thanks. I wasn't chasing for compliments. I was actually wondering if you had something there. Okay, so I think one point of the contention, I think we're all on board with, you know, we do some sort of a pre-training here on GitHub. We do some sort of a fine tuning on the problem we're interested in, right, which is these coding problems. But then I think the point of contention that a lot of people have is this sort of this approach of large scale sampling followed by filtering, which is really different than how a human solves problem. This is I'm as a programmer, I don't I don't blast out 100,000 different possible solutions and then, you know, run them all, not even in my mind, right? Not even that's not even the way I think to sort of sample forward and then test all of these things. I'm actually impressed that this, you know, that the filtering step would would give you the sort of the correct things right here. So my, my question would be, I'm willing, let's say, to, to disregard the fact that that's not mechanically how I do it. I'm willing to still consider the possibility that the model will actually, you know, given the attention maps and so on actually does, you know, do something worthwhile more than just kind of random sampling, right? Because if I were just to random sample, I would never get a solution. So I'm willing to see that the model might be doing something. And then I thought, well, if that's the case, shouldn't I somehow find a representation of the abstract concepts inside of the latent spaces somehow, you know, when whenever the algorithm is about sorting lists, shouldn't I find like list primitives and sorting algorithm comparison operators and something like like the concepts that I would think of when implementing this algorithm, or like a Dykstra's nearest neighbor algorithm? If I if I implement that, shouldn't I find these things? Have you thought of like investigating the model and see whether or not it kind of learns programming concepts by itself? Is that even, you know, possible? I mean, that's a very interesting question, right? We've done a lot of analysis on the model. But as we report in section six of the paper, it's either centered on the impacts of the end metric, like the solve rates, or we analyze the sample themselves. And Peter's done a great job, by the way, showing that our models don't really copy paste. But we haven't yet prodded the model enough internally to be able to answer that question definitively. If I had to venture a guess, though, I'd say it's very likely that these concepts are present at the latent space level. And as you just said, the best proof of that is that the model does actually come up with these relevant concepts and implements them to solve some of the problem, right? So we have tree traversals, we have dynamic programs, we have sorting, all these sort of things. So they're definitely there. It seems to me very likely that they're here. And yeah, doing massive sampling alone cannot explain the solve rate that we have. I think another issue, though, is that probably the right concepts are there, but they're in there amidst many, many other concepts. And picking exactly the right concept at the right time is actually really difficult. Yeah, I think I'd probably add something to that, which is, I guess, maybe the last point that Remy made is not even specific to the transform work that we have. When I read a competitive programming problem, I've got five ideas in my head of what might work. So I think that wouldn't be that bad, even if there was a bunch of different things in there. One other thing I think I'd add is that, I guess, because we sample from the model autoregressively, the latents are actually changing as you do that. And so later on, the model may not have honed in on the concept of, oh, I need to do a DFS here, or I need to do Dijkstra's algorithm until maybe 50%, 80% of the way through the problem. So I think if we were to do that investigation, we'd have to consider how that changes through the sampling procedure. It's not even clear where to look, basically. Is it at the end of the encoder? Is it during sampling? We don't know. Yeah, it is also, I mean, it connects to this larger problem of people arguing whether or not these models can, quote unquote, reason, right? And you explicitly in the paper also make an effort to connect this to abstract reasoning and so on. I think, you know, investigating things like this here could be sort of a proxy for really demonstrating, yes, there is actually something in these models that amounts to sort of symbolic abstract reasoning, even though we do sort of next token prediction. So yeah, I think it's fairly cool. I guess, can I jump in there? Yeah. So I was just saying, like, one kind of more general point there, I think, is that, you know, I definitely see this as, it's like clearly different from how I solve a problem. But also, I think in machine learning, like, maybe, you know, the first step to doing something the right way is doing it at all. And I think that's kind of, you know, part of what we've achieved here. Do you have plans to bring down this large scale sampling? Like is there any ideas floating around of, you know, maybe we don't have to sample a million things and then test them all? I mean, I think, of course, it would be somehow more satisfying if our model could just like one shot the problems. And I think getting higher quality average samples is a really interesting research direction, especially since, yeah, every time you want to solve a problem, you probably don't want to have to try and begin different things, right? That's typically not how we work. But I think there's also something really interesting in this scaling that we observe, and the fact that we can actually get more and more good answers by simply by something more is something that's quite interesting to explore. And what's further interesting, I think, is that the larger, like the model size seems to be also correlated with the quality of the samples in itself, which is also something I find cool. Yes, indeed. We see that the bigger the model, the higher we start and the steeper the slope basically in the sampling curves. So on average, the bigger the model, the better the sample quality. A lot of models have popularized or a lot of systems in recent times have popularized this idea of sort of having an additional model to do filtering output of generative models, right? Most famously, I guess, Dali, which uses the clip model to sort of rerank or filter the outputs. You here have a rather, let's say, heuristic way of filtering the outputs. Is it even possible or considerable that you would sort of train another model? Or would that just shift the problem? I'm going to guess, you know, if training a model that can tell me whether a program is correct for a given solution, that's almost like solving the problem itself. But you know, we've seen that it generally helps to pair generative models with rankers. Is that something that is in scope here? Or is there a particular reason why that wouldn't work? I think that's a very reasonable suggestion. And over the course of the project, we've tried several ideas that are linked to this, particularly training value functions, which could be used either as guides during the sampling process or as a ranking mechanism once the sampling is done. What we've found, though, is that learning a good enough value function remains extremely challenging. And so we're definitely interested in trying these ideas again. It's just that we haven't been able to make them work quite yet. And why that is, is still a bit up for debate. Of course, we have a rather small functioning data set, which might be part of the reason why, or maybe the action space is too big. We are still investigating that. Yeah, I wanted to add something to that as well, which is that I think, yeah, we definitely tried to re-ranking a couple of times, and it seems like a good thing to try. But the way that we eventually did a lot of that filtering was by executing the program. And that is an enormous boost. And I think whether we had a ranking model or not, we would definitely still do that. And there are ways of using the program execution that we haven't even considered. We just use the fact that the public test passes or doesn't pass. So I think potentially even continuing to use that or even expanding on how that happens, how executing the program affects the filtering and ranking is also another kind of interesting, I guess, non-machine learning way to continue doing that. I'm all for non-machine learning. I'm all for not introducing more models. But you do point to a good question. There is this small set of candidates, which comes from these large sets of potential solutions. And the filtering is a really important step there. As you say, you execute the programs against a small set of samples. Now this set is maybe four, maybe five test cases or something like this. And I haven't seen, maybe I've overlooked that, but I haven't seen anywhere in the paper where did you investigate if we had 10 such public test cases, how does that change? Or if we just had one, how does the success of the model change with the amount of test cases you have at your disposal in the given problem? That's actually a really good suggestion. We haven't looked at that. I think in the end, the issue for us is we don't really have control over this quantity. And most problems have very, very few public test samples, between one and three on average, I think. So we didn't really push this direction because we thought we can't move the needle on it at test time. But that doesn't mean that it wouldn't be informative to try to see. And if I had to take a guess, I would imagine that adding more public tests would be very helpful because it would make the filtering mechanism that much more powerful. So yeah, that's basically how I think about this. And of course, we could try to generate more tests, but that's a very difficult problem in and of itself. Yeah, I think I had another thought on that, which is that I actually would love to do that ablation, but actually not necessarily for the problem that we had, because as Remy said, we can't control the number of public tests we have. But there may be some applications of something like AlphaCode where you can control the number of public tests, and knowing how that affects the ability of us to filter the samples would be super interesting. Maybe two samples is enough to get you exactly the right solution most of the time. Unit tests come to mind, right? Just programming essentially by writing four or five unit tests for a function or a class that I want to write, and then just let the model come up with a bunch of examples for me to choose. Yeah, I think that would be, I don't know, like the future of programming looks more and more something I don't recognize from that, I think is very exciting. Is there some sort of, you know, between these two, is there some sort of adversarial setup that I could do? You have various models, like you have a model that generates new test cases, but at various stages, right? So for the clustering, you simply need to execute and observe the same outputs. Because I'm going to guess a model that makes new test cases doesn't necessarily make correct test cases. But is there also a model that makes test cases just sort of generates them, let's say, in a language model way, in a, you know, most likelihood way? Do you ever think of some kind of adversarial setup, given that DeepMind is a lot of in the space of like self play and sort of this reinforcement learning setting? Is there opportunities here for sort of systems to challenge each other to get better? Yeah, that's, it's very funny that you mentioned that because the project started off right after the AlphaStar project, basically. And so we had our minds were full of these types of ideas. Right. And so that's something that I've actually been very keen on since the inception of the project more than two years ago, to bring some notions of self play, curriculum learning, etc. I think that that would be very exciting. Unfortunately, generating new problems is an extremely difficult task, because first of all, your problems need to make sense. They need to actually be solvable. Right. So I can definitely see a world where we have many, many problems. And either they're way too difficult or they're nonsensical. And the other thing is we also have to come up with unit tests that work with the description of the problem. Right. And we have we have a data set of 12 to 13,000 problems, if I remember correctly, which is probably not enough for us to train a really good generative model to ask problems. So we haven't, we haven't really tried up until now. So I guess maybe I think one distinction I think is relevant there is that in AlphaStar and in a couple of other self play setups, they are symmetric. So you kind of expect the both sides to be improving all the time. Whereas in our case, it's less obvious how you might improve the problem maker over time. Maybe there is a I have no clue how these problems are actually made because humans need to make these programs. Right. If I look at a problem problem description like this, I'm like, this is this is insane. Not only is it very thorough, right. Also I have to somehow make sure that I as a maker of the problem don't make a mistake. And when I generate test cases, usually, you know, the example inputs right here are kind of small, but then I need to test like all the edge cases, right, to make sure that people have the correct algorithm, which means some are going to be very long and so on. So I almost have to write like a generator for, you know, these these long things. Maybe there isn't maybe there's a way to replicate that process of like how humans come up with these problems as because they're going to have like strategies and whatnot. They just they don't just sit there and go like, well, backspace. Right. I don't know, have you looked into do you know how these problems are made, like on a mechanical level? So I think we've been focusing a lot on the solving aspect of things and a lot less than the generating problems aspect of things. I have I have a healthy respect for the difficulty to generate problems that people can actually solve. Right. So I think we've been doing exams and thinking this is no fun. And then I know a lot of people who are teachers who have to actually devise exams. I think, wow, this is even less fun, actually. But yeah, I don't think we have a really good grasp on the human generative process for this thing. It would be really interesting to discuss with problem makers to see what are the strategies and whether or not we can try to replicate that and when possible direction would be to actually help them. That would be quite cool. Yeah, I think that's sorry. I think that's a great idea, actually. Like I I'm really quite interested to go and ask them myself now, I think. Maybe like if I had to do I would look in a computer science textbook and for like algorithms and then dress them up in some kind of story. That seems to be like what what a lot of problems are. But yeah, in terms of doing it mechanically, maybe that would be even harder than generating the solutions because like lots of people upload their solutions to GitHub. But I guess I expect there would be less data on how to create problems on. Yeah. Yeah, I was I was exactly I was more thinking of there must be some process because also these these people have to come up with new and new problems, right. And there's only so many algorithms and something like this backspace problem. It's very intricate, right? There is not really like an algorithm that I can just poof apply like I really have to think through stuff. One of my questions is that you hear the test cases, the public test cases, they're kind of samples, right? For you also to think through as a human. But very often, the testers, they also want to test not only whether you have the correct algorithm, but also whether you have the sort of correct runtime algorithm. Because you know, I can write an algorithm, you know, in I don't know, like if I have an O of n squared, that might not be the algorithm the tester is looking for. So they want like the O n log n. I'm having trouble writing the O n log n algorithm, right? Because one is really easy to implement. And one is actually the challenging one. So they will make deliberately like very large hidden test cases, so that my my naive algorithm would either go out of memory or out of time on the evaluation server. And this is something that you would not capture with just filtering on the public test cases as as your algorithm does. Your algorithm would think, well, I've solved the problem, right? I've come up with a solution. The naive solution will probably even be the more likely one given the language model. And then right and then it's it's filtering, it's clustering is like, well, all of this seems just fine, right? How do you have any grasp on how good you are on these types of problems? And is your model does it have some strategy to overcome that? Yeah, I think I can take that. The main answer here is that we just don't we just don't do it. We when we actually like looking at what our real self rate is, we had to do a lot of manual checking of solutions to check that they were meeting asymptotic complexity requirements of that we expected the problem to actually have. I think you do you mention before the call or in your question about clustering to buckets by by time or memory, I think you wrote that down. Did you have this in the paper or was this something I came up with? I don't I don't think that you came up with. Okay, yeah. Yeah, is this I mean, is this is this viable or is this like a bad idea? Or? Yeah, I guess I just had a thought on that. I think it's quite a cool idea. Maybe that particular implementation of looking at time and memory usage of of inputs like definitely is in the theme of, you know, executing the program and saying what happens. So I think an idea along that lines is is actually worth a go. One thing I would say is that a lot of these problems, I think, when you write the solution, which is asymptotically better, usually has like a big constant factor in front of it or a constant additive complexity. So you'd have to kind of consider that and whether that is going to adversely affect which solutions you're removing, maybe you're removing the thing which actually is going to have actually the asymptotic complexity. I think we could probably use it to cluster, right? Because then we had different if you had the same different asymptotic implementation, you would have different different values. But choosing directly according to like trying to rank them, depending on the performance on very, very small unit tests, we would probably I mean, my intuition. And our intuition, I guess, is is that we'd have to be extremely careful how we do that and not to overfit too much to that particular metric. So something that I want to point out, though, is that, yes, sometimes we have what we call slow positives, which are correct, except that they're impractical. But still, I already find that to be quite impressive, because some of these problems we go for the naive approach, but it's not completely evident that the naive approach would even work. So there's this thing like you want to remember, coding mentor told me about just make it run, make it right, make it fast. So we make it run, we make it right. Now all we have to do is to make it fast, which admittedly is a really difficult problem. I think I wouldn't be too worried that the clustering might not work. I would be more worried that the language model itself might not even, you know, might just jump on the sort of more likely naive implementation and never actually get to output the very different, possibly more efficient implementation, because these two things, they don't often look similar. They often look very, very different from each other. And yes. I think another issue is in our pre training sets on GitHub open source code, probably very, very fast, efficient programming isn't the majority of what's on there. So it might be that there's a bias towards simpler, more naive solutions already when we start fine tuning. So of course, we'd have to fight against that. With respect to the sampling and whether or not you can output something, you have a lot of tricks to increase your sampling diversity. One of the most notable things is that you have this prefix right here, which I found quite quite genius. I think in general, the approach of including sort of unknown things like that you would only know at training time, like things about your labels into the prompts, and then having that as sort of like a dial where you can control the model. I think that is a very cool, very cool idea. And I think you've shown quite quite impressively how that can help. You use it mostly to use it to to vary the outputs of your model. But that brings me like, given that we have to do all of these things to increase diversity, do you think maybe where our sampling procedure as such isn't a very good one? Because we have to do all these tricks, like could we fundamentally remake our language models or our generative models to to be more like diverse, let's say? Yeah, so I do think you're right. And we're not equipped with the right tools just yet. Right now we have this very crude setting to tune, which is a sampling temperature. But this means that we have very little control over how qualitatively diverse our samples are going to be. All right, so we're searching over the model distribution in an extremely crude way, which is basically pointing it into a general direction and say, OK, try to take as many sample ports as you can in that particular direction. But it seems important to me that we should be able to branch out in different directions only at fairly select decision points, not on every step. And we don't have a proper mechanism to do that. So we have high hopes for top K and nuclear sampling or for our sampling being guided by a value. But as we report in paper, this didn't really bring significant improvements. And I think another thing here is that we are sampling very independently. We're not taking past samples into account. When sampling a bit more autoregressively at the level of samples could probably be an interesting thing to explore. Yeah, I had one other point there. Since we sample from the models autoregressively, maybe this isn't really related to the diversity point, but to something in general, that's clearly not how I do things at all when I'm writing code. I usually write something, I write a sketch, and then I iterate over it in random bits of the code. So it's possible that that also is something that needs to fundamentally change by the way that we sample from models. I haven't looked much at the outputs the model generates, which astounded me. Just seeing this and seeing it output from a language model is astounding by itself. But also, it's very instructive. On the right, you even do a little bit of analysis and say, you know, these lines are this, these lines are this, these lines are this. Did you generally find that throughout your solutions? I haven't looked at many more solutions, to be honest. Did you generally find that code is interpretable, you know, very, very sort of instructive? Or is this a particular problem that you've picked out and to show kind of like, oh, look, the model solves the problem in an understandable way? Or did you, was most of the output cryptic or understandable? Yes, I think I looked at a fair few, you know, individual solutions when I was doing the analysis for this paper. I think in general, so actually, to be clear, like we did definitely pick this example as something that, you know, illustrates what's going on. But in general, you know, the model does produce things which you can read and understand what's going on. I think you have to, you know, and that's kind of expected in a way because we're training on human data, right? We're training to mimic the way that human programs look. So that's not crazy. But when we fine tune, competitive programmers write very unreadable code. So that's another thing to bear in mind. They will use a lot of type devs in C++, for example, a lot of crazy helper functions. And that's also something you see a lot in some of the solutions. You'll see these like huge copy pastes of code which like passes an input in an efficient way. A lot of that is dead code and it doesn't actually get used. And that's consistent with some of the competitive programming, like real solutions. But yeah, I guess like in this, you know, maybe it's because we filter for public tests as well, like in particular, the solutions which are correct seem to be fairly interpretable and make sense. But yeah, on rare occasions, like the implementation is quite difficult to understand. But yeah, I think if you want to look into that a bit more, we do have the tool, alphacode.dmin.com, which Remy and Julian worked on. And there's also some commentary on there, I think, from Petr, who works at Google, about what the model is doing. And I think in the samples he looked at, generally, he was quite happy that a lot of them seem to be doing something that you would expect in a reasonable way. I mean, it's distantly possible that you write something that just passes all the test cases but isn't actually correct. Like we're sampling so many things, like this might be not very likely. So it's definitely possible. And we did a fair amount of work actually generating new tests to try to make sure that that didn't happen. I remember somewhere, maybe a little bit under a year ago, we took a deep dive on our solved rate and we were trying to figure out whether it was the actual thing or whether actually we were gaming the problems. And we realized that there was a significant percentage of our solutions, quote unquote, which were getting the system. And the possible reasons for that were that actually there was very little coverage because there were many tests, but the answer was always the same. Sometimes you have yes, no type of things. And you look at the private test and the answer is always yes on the 40 private tests. And so the model will try, if you sample from it a million times, it will try to just print yes. That's probably going to happen. And for other things, we just had very, very few tests. So we filter out the problems, we had too few tests, but we also mutated the tests to add new ones to make sure that this didn't happen. And I think we went down from, I don't remember if it was 40% or maybe even 60% false positive rates to about 4% in our final data set, which is still significant, but we've found that was a reasonable and acceptable amount of false positives. I don't think I mentioned this in the video too much, but you have this kind of fuzzing approach to generating new test cases where during training, you know the correct solutions. So you can essentially generate new correct test cases by using the correct solutions that you know are correct, which I found, yeah, it makes sense. I think in this space of programming, you can do a lot of these things, which is neat. So what happens basically is we mutate programmatically the inputs of the tests that we already have, and then we run the human correct solutions on them. And then if we filter these new mutations, because some of them might not actually be correct inputs, and we figure out whether the human solutions actually agree on an output. And when we have a sufficient level of agreement on a given output, then we add this mutated input to the output that's generally agreed upon. Now, you mentioned before that you had high points and low points during the process of this project. Again, I can imagine that might be one of the lower points when you realize, wait a minute, all we do is false positives. Could you, I don't know, could you let us in maybe on what was sort of the lowest point? Was there a moment where you thought, ah, this isn't going to work out, you know, after all this time? And what did you do to overcome these things? That's a tough question. When was I think the lowest point probably wasn't the same for all the members of the team, right? I think we did, because we were working on slightly different ideas most of the time. But I think there was in the middle of a project, there was basically a month where we had very, very little progress. And so we had these meetings every week when we would see what was the best performing thing and it was still the same thing. So there's that, that was definitely no point for us. And maybe like also when some of the big ideas that we thought were going to help didn't pan out. Like for instance, when we realized that for whatever reason, it was just too hard to train a really good value function and we weren't going to be able to leverage all of the methods that this would have unlocked, which we did rely upon at least initially in our main map. So yeah, that would be my answer. I definitely had a couple of those myself. But I think in general, a lot of the times we realized that we got results which weren't actually true because they were false positives. Later on, we did claw back a lot of the gain. But I think that's just maybe the scientific method at work. We kind of proved us, we tried something and then we realized actually it wasn't working. But yeah, I think having our metric to guide us there really helped us get through those. I think we were well served by a somewhat skeptical approach when we had a result that looked good to be true. Our initial thought was okay, this is good to be true. Where's the issue? And more often than not, there was actually a bug that we found. Once you released the, let's say the paper and so on, I think a lot of comments started coming in. Did you have a criticism that, what is the most valid criticism that you've encountered that you didn't foresee? Obviously, you have a lot of limitations at the end of the paper and you make it very clear like this is one niche, this is this, there's limitations here. Is there something that people brought up and you were like, oh yeah, I didn't think of that. That's a good point. There's a few things, it's a difficult question generally, but there's a few things definitely. Generally, as we said, we've been very happy with how the work was received and we've gotten a lot of constructive feedback. Dima Badanoff's Twitter thread is a good example, for instance, where he outlined why he thinks and we do agree with him that we're still a long way from top level human performance on this task. I was also made aware that the data that we put on alphacode.deepmind.com was actually not correct. I had filtered the correct solutions wrong. So again, underlining the importance of doing that right. So I thank everybody who told us, well, I don't understand this correct solution. It's actually not correct. And they were right. So now we've fixed that. So if you go to alphacode.deepmind.com, you will get actually correct solutions. And then something that surprised us, but I don't know whether it's valid or not, is that a fair amount of people seem to think that the average human competitor on codeforces.com is not very good, which I think we have a fairly different view. So I'm not sure I would say it's valid, but it was certainly surprising to us. And then in terms of the limitations of the model, we thought a lot and just a bit of what we thought were the weaknesses. So I'm not sure that I've seen anything that we hadn't already identified. Cool. Where do you see this more in the real world? We talked about programming, competitive programming, maybe a future where I can just write a bunch of unit tests and this will go fine. But there are obviously applications beyond this. Are there people maybe in your team that are already eyeing or maybe you have some ideas of this? Where could this be used outside of programming? Just the techniques in here and the methodologies. Do you see some sort of semi-obvious transfer to a real world problem other than coding? I think generally speaking, there's going to be a lot of downstream applications for general purpose problem solving AIs. To our team, we've been thinking a lot about programming and less about non-programming applications. So I think Farfakir, there's some natural directions, which include developing tools to make coding easier, as we already touched upon with automated test generation, smart autocomplete, etc. Or maybe tools to make it easier to learn how to code. So you could imagine an AI that can comment and suggest some improvements to your code, etc. But I think the applications that could be used to democratize programming are definitely on our radar. In terms of applications not directly related to programming, I haven't thought too much about that. I'm fairly certain that problem solving is sufficient in general so that we will find interesting applications, but we haven't been too much on the lookout for that. I think you're right to point out a couple of those ideas, Yannick. And I think Codex has also shown us that this works. You can build a product out of these kinds of models, and people are really happy with it. So it's definitely something that we're thinking about, but I think we definitely haven't concretely made any decisions at all or finished brainstorming even, whether that's something that we'd like to do. But yeah, I think maybe to go back to one thing that Remy mentioned earlier is that the methods that we use are actually pretty general, I find, as far as programming goes. The filtering, which is the really big one, could definitely be used in an application. But a lot of what softwrench does is just nothing to do with writing code. And one way I guess I would think about it is what we've done is take a description of a problem and actually a complete description of a problem and map that to code. But really, I find in my day-to-day, I'm spending maybe 50% or more of my time talking to people and writing that description, if that makes sense. Yeah, Alpha requirements engineer is the next paper. Is there anything else you want to get out about this paper? Can people somehow get started with or get into this type of research or anything you'd want to communicate? I think we'd be really excited for other researchers to work on this. I know some other researchers are already working on this problem, but our goal is that as many as possible actually work on this problem because any gain we make here is going to be distributed. So that would be really nice. And that's why we released our data set, which we spent a fair amount of time on and we think is a really good tool to approach these problems. As we showed in the paper, you don't need huge models to actually start solving problems. So you can do that with less resources. Of course, there's the issue of having to sample a whole lot, but I would say that's a very exciting research direction to actually reduce the amount of samples you have to take to solve these problems. Peter, any messages for anyone listening? I think as Remy said, the fact that we released the data set is clear that that's the main point that you should start. But I think in general, I'm optimistic not just about competitive programming, but about people working on programs in business in general with machine learning. So I can only encourage people to go and do it. And actually, I should say that as a programmer myself, I'm quite optimistic that working on this kind of problem is going to make my life a bit easier. In this case, Peter and Remy, thank you very much for being here. This was a lot of fun. I learned a lot. And I hope to see the alpha requirements engineer in the future. Thanks for having us.
 Hey, this is an interview with the authors of the Alpha Code paper by DeepMind.
 This is a crazy system.
 It does automated competitive programming and is about as good as an average human in
 real competitions, which is crazy.
 In case you haven't seen it, I've made a comprehensive paper review of this paper in the last video.
 So be sure to check that out because the authors that I'm interviewing today have also seen
 that video and were able to dive right into the matter answering any questions, any criticisms
 and so on.
 You're also able to get a behind the scenes look into what things went wrong during this
 research, things that didn't work out, things that were red herrings and much more.
 We also talk about how the project came to be and how the authors dealt with the immense
 media reaction that followed the release.
 Let me know how you like these types of videos.
 Having the authors on is a huge privilege and I'm absolutely sure you'll learn something
 useful from this conversation.
 If you like content like this, don't forget to leave a like, subscribe, tell me what you
 think in the comments and I'll see you around.
 Bye bye.
 Yeah, hi everyone.
 Welcome back.
 I'm here today with Rémy LeBlanc and Peter Choi, who are authors of the competition level
 code generation with Alpha Code paper.
 I'm just going to call it the Alpha Code paper.
 Everyone's excited about this paper.
 So much hype around it and it's very cool to have the authors with me.
 So Rémy and Peter, thank you very much for being here.
 Thanks for having us.
 Thanks a lot for having us.
 Yeah, we're quite happy to be doing this with you today.
 So the paper, obviously, given that the machine learning community and the programmer community
 intersect in large parts and then the competitive programming scene also is kind of known for
 not being the most humble.
 Obviously, let's say, there was quite a bit of hype, quite a bit of media reception around
 the paper.
 Did you expect anything like this and how did you experience sort of how the paper was
 received in public?
 I guess I can take that one for a start, Peter.
 So I think overall, we've been fairly happy with how the paper has been received, right?
 People have been talking a lot about the ideas that we put forward and the results that what
 we think is fairly impressive for what we're trying to do is nowhere near what might have
 been reported in some news outlets.
 So we did expect that there was going to be positive reactions, negative reactions and
 a bit of misunderstandings, probably.
 But I think overall, we've been fairly happy.
 Yeah, I think we spent a few hours, maybe even a day or two after we released the paper,
 just kind of watching with popcorn what was going on.
 And yeah, that was pretty enjoyable.
 But yeah, overall, I'd say I'm pretty pleased.
 Do you want to maybe just as an opportunity to...
 Did you hear like crass overstatements you said, you know, some people said a bit more
 than what you actually did.
 So is there something that you saw that was like really where you say, no, this is actually,
 this is wrong.
 It's too much, you know, rather than just selling it very prettily.
 Anything you sort of want to bring down to earth.
 I think I can definitely add one thing there.
 I think the biggest thing that I noticed and like quite a common mistake was to like overstate
 our result as DeepMind, you know, has an algorithm which is as good as an average programmer.
 But like really, the right answer is, it's average competitive.
 You know, we get the same results as an average competitive programmer.
 And those are like huge, huge, there's a huge difference there.
 But you know, that distinction can be like a bit nebulous if you're not familiar with
 the programming or competitive programming.
 So that's the one, the main thing I think which becomes the top of my list.
 Yes, of course, like most of the most of your job as a software programmer isn't actually
 writing code, right?
 It's reading code, understanding code, thinking about how to achieve whatever it is you want
 to achieve, right?
 So we focus on a much, much narrower scope in this paper where we have a very precise
 description of what we want to do.
 We have examples, we have constraints, etc.
 Which to us is a very interesting proxy for problem solving.
 But it's very far from the full job of an actual developer.
 Yeah, I was, I mean, I was, I think even with the with the correcting the record, it is
 still very impressive.
 And I think before we before the recording, we talked about that also you seem to have
 been a bit surprised at how far you were able to get with this system.
 Could you tell us a little bit about the just the process of, you know, how did you start
 out?
 What did you do?
 For example, codecs or copilot from GitHub.
 And I have to say it's like is really good.
 Like it's, I think it's it's a game changer if the UI is cleaned up a little bit and models
 like this will be, you know, I think assisting programmers a lot.
 But how did you go from like that?
 Were you even aware of codecs copilot?
 And how did you get to to alpha code?
 And what did you expect?
 Right, so I think and I mean, I wasn't there from the very beginning of the of the problem.
 But I think we've always been focusing on a slightly different approach than what codecs
 and copilot are doing.
 I think we're really interested in this aspect of problem solving and we were really interested
 in this aspect of generalization.
 We wanted to solve unseen problems and come up with novel solutions to things that the
 model hadn't seen during training.
 And so competitive programming was sort of a natural target for us.
 And then we started getting a bit of traction and we set ourselves what we thought to be
 almost an impossible goal.
 But we thought we needed to be ambitious to really, really push ourselves and push the
 push the methods.
 And so our level of confidence in whether or not we're going to achieve this fluctuated
 during the course of the project.
 At some points we had high points and we had low points.
 Some points we're convinced we're going to succeed.
 At some points we had pretty severe doubts.
 But yeah, in the end, we managed to get all the way across the finish line.
 I think one thing I'd add to that is I think this is the first project where I worked on
 which had quite a strict adherence to looking at a particular metric quite regularly.
 And I think that really helped us incorporate ideas that were happening, that were being
 researched within DeepMind and outside of DeepMind.
 So I think that was really worthwhile and something that we've learned to value quite
 a lot in working on these ambitious projects.
 It's cool if you have some sort of a North Star, right?
 At least you know where you want to get.
 I think with most projects it's even ill-defined kind of where the end goal is.
 And I think it's probably half the game in academia and also projects as such.
 So I've made this little overview and intro to your paper.
 Did you feel that was accurate?
 Is there anything missing?
 You want to amend on how the system works?
 Any wrong emphasis that I've set?
 I don't think there's anything wrong with what you described.
 And I was fairly impressed that you managed to sort of distill this massive paper down
 to a reasonable size in terms of the video.
 So yeah, I think I was quite happy with the way you described it.
 Of course, opportunities to get into more details by reading the paper itself, especially
 on the maybe on the method section.
 But overall, it was really good.
 I was really impressed as always.
 Yeah, I generally love your videos, Yannick.
 So it's a really easy way to get an overview of a paper and decide if you want to read
 it yourself at all.
 And yeah, this was kind of not an exception.
 Thanks.
 I wasn't chasing for compliments.
 I was actually wondering if you had something there.
 Okay, so I think one point of the contention, I think we're all on board with, you know,
 we do some sort of a pre-training here on GitHub.
 We do some sort of a fine tuning on the problem we're interested in, right, which is these
 coding problems.
 But then I think the point of contention that a lot of people have is this sort of this
 approach of large scale sampling followed by filtering, which is really different than
 how a human solves problem.
 This is I'm as a programmer, I don't I don't blast out 100,000 different possible solutions
 and then, you know, run them all, not even in my mind, right?
 Not even that's not even the way I think to sort of sample forward and then test all of
 these things.
 I'm actually impressed that this, you know, that the filtering step would would give you
 the sort of the correct things right here.
 So my, my question would be, I'm willing, let's say, to, to disregard the fact that
 that's not mechanically how I do it.
 I'm willing to still consider the possibility that the model will actually, you know, given
 the attention maps and so on actually does, you know, do something worthwhile more than
 just kind of random sampling, right?
 Because if I were just to random sample, I would never get a solution.
 So I'm willing to see that the model might be doing something.
 And then I thought, well, if that's the case, shouldn't I somehow find a representation
 of the abstract concepts inside of the latent spaces somehow, you know, when whenever the
 algorithm is about sorting lists, shouldn't I find like list primitives and sorting algorithm
 comparison operators and something like like the concepts that I would think of when implementing
 this algorithm, or like a Dykstra's nearest neighbor algorithm?
 If I if I implement that, shouldn't I find these things?
 Have you thought of like investigating the model and see whether or not it kind of learns
 programming concepts by itself?
 Is that even, you know, possible?
 I mean, that's a very interesting question, right?
 We've done a lot of analysis on the model.
 But as we report in section six of the paper, it's either centered on the impacts of the
 end metric, like the solve rates, or we analyze the sample themselves.
 And Peter's done a great job, by the way, showing that our models don't really copy
 paste.
 But we haven't yet prodded the model enough internally to be able to answer that question
 definitively.
 If I had to venture a guess, though, I'd say it's very likely that these concepts are present
 at the latent space level.
 And as you just said, the best proof of that is that the model does actually come up with
 these relevant concepts and implements them to solve some of the problem, right?
 So we have tree traversals, we have dynamic programs, we have sorting, all these sort
 of things.
 So they're definitely there.
 It seems to me very likely that they're here.
 And yeah, doing massive sampling alone cannot explain the solve rate that we have.
 I think another issue, though, is that probably the right concepts are there, but they're
 in there amidst many, many other concepts.
 And picking exactly the right concept at the right time is actually really difficult.
 Yeah, I think I'd probably add something to that, which is, I guess, maybe the last point
 that Remy made is not even specific to the transform work that we have.
 When I read a competitive programming problem, I've got five ideas in my head of what might
 work.
 So I think that wouldn't be that bad, even if there was a bunch of different things in
 there.
 One other thing I think I'd add is that, I guess, because we sample from the model autoregressively,
 the latents are actually changing as you do that.
 And so later on, the model may not have honed in on the concept of, oh, I need to do a DFS
 here, or I need to do Dijkstra's algorithm until maybe 50%, 80% of the way through the
 problem.
 So I think if we were to do that investigation, we'd have to consider how that changes through
 the sampling procedure.
 It's not even clear where to look, basically.
 Is it at the end of the encoder?
 Is it during sampling?
 We don't know.
 Yeah, it is also, I mean, it connects to this larger problem of people arguing whether or
 not these models can, quote unquote, reason, right?
 And you explicitly in the paper also make an effort to connect this to abstract reasoning
 and so on.
 I think, you know, investigating things like this here could be sort of a proxy for really
 demonstrating, yes, there is actually something in these models that amounts to sort of symbolic
 abstract reasoning, even though we do sort of next token prediction.
 So yeah, I think it's fairly cool.
 I guess, can I jump in there?
 Yeah.
 So I was just saying, like, one kind of more general point there, I think, is that, you
 know, I definitely see this as, it's like clearly different from how I solve a problem.
 But also, I think in machine learning, like, maybe, you know, the first step to doing something
 the right way is doing it at all.
 And I think that's kind of, you know, part of what we've achieved here.
 Do you have plans to bring down this large scale sampling?
 Like is there any ideas floating around of, you know, maybe we don't have to sample a
 million things and then test them all?
 I mean, I think, of course, it would be somehow more satisfying if our model could just like
 one shot the problems.
 And I think getting higher quality average samples is a really interesting research direction,
 especially since, yeah, every time you want to solve a problem, you probably don't want
 to have to try and begin different things, right?
 That's typically not how we work.
 But I think there's also something really interesting in this scaling that we observe,
 and the fact that we can actually get more and more good answers by simply by something
 more is something that's quite interesting to explore.
 And what's further interesting, I think, is that the larger, like the model size seems
 to be also correlated with the quality of the samples in itself, which is also something
 I find cool.
 Yes, indeed.
 We see that the bigger the model, the higher we start and the steeper the slope basically
 in the sampling curves.
 So on average, the bigger the model, the better the sample quality.
 A lot of models have popularized or a lot of systems in recent times have popularized
 this idea of sort of having an additional model to do filtering output of generative
 models, right?
 Most famously, I guess, Dali, which uses the clip model to sort of rerank or filter the
 outputs.
 You here have a rather, let's say, heuristic way of filtering the outputs.
 Is it even possible or considerable that you would sort of train another model?
 Or would that just shift the problem?
 I'm going to guess, you know, if training a model that can tell me whether a program
 is correct for a given solution, that's almost like solving the problem itself.
 But you know, we've seen that it generally helps to pair generative models with rankers.
 Is that something that is in scope here?
 Or is there a particular reason why that wouldn't work?
 I think that's a very reasonable suggestion.
 And over the course of the project, we've tried several ideas that are linked to this,
 particularly training value functions, which could be used either as guides during the
 sampling process or as a ranking mechanism once the sampling is done.
 What we've found, though, is that learning a good enough value function remains extremely
 challenging.
 And so we're definitely interested in trying these ideas again.
 It's just that we haven't been able to make them work quite yet.
 And why that is, is still a bit up for debate.
 Of course, we have a rather small functioning data set, which might be part of the reason
 why, or maybe the action space is too big.
 We are still investigating that.
 Yeah, I wanted to add something to that as well, which is that I think, yeah, we definitely
 tried to re-ranking a couple of times, and it seems like a good thing to try.
 But the way that we eventually did a lot of that filtering was by executing the program.
 And that is an enormous boost.
 And I think whether we had a ranking model or not, we would definitely still do that.
 And there are ways of using the program execution that we haven't even considered.
 We just use the fact that the public test passes or doesn't pass.
 So I think potentially even continuing to use that or even expanding on how that happens,
 how executing the program affects the filtering and ranking is also another kind of interesting,
 I guess, non-machine learning way to continue doing that.
 I'm all for non-machine learning.
 I'm all for not introducing more models.
 But you do point to a good question.
 There is this small set of candidates, which comes from these large sets of potential solutions.
 And the filtering is a really important step there.
 As you say, you execute the programs against a small set of samples.
 Now this set is maybe four, maybe five test cases or something like this.
 And I haven't seen, maybe I've overlooked that, but I haven't seen anywhere in the paper
 where did you investigate if we had 10 such public test cases, how does that change?
 Or if we just had one, how does the success of the model change with the amount of test
 cases you have at your disposal in the given problem?
 That's actually a really good suggestion.
 We haven't looked at that.
 I think in the end, the issue for us is we don't really have control over this quantity.
 And most problems have very, very few public test samples, between one and three on average,
 I think.
 So we didn't really push this direction because we thought we can't move the needle on it
 at test time.
 But that doesn't mean that it wouldn't be informative to try to see.
 And if I had to take a guess, I would imagine that adding more public tests would be very
 helpful because it would make the filtering mechanism that much more powerful.
 So yeah, that's basically how I think about this.
 And of course, we could try to generate more tests, but that's a very difficult problem
 in and of itself.
 Yeah, I think I had another thought on that, which is that I actually would love to do
 that ablation, but actually not necessarily for the problem that we had, because as Remy
 said, we can't control the number of public tests we have.
 But there may be some applications of something like AlphaCode where you can control the number
 of public tests, and knowing how that affects the ability of us to filter the samples would
 be super interesting.
 Maybe two samples is enough to get you exactly the right solution most of the time.
 Unit tests come to mind, right?
 Just programming essentially by writing four or five unit tests for a function or a class
 that I want to write, and then just let the model come up with a bunch of examples for
 me to choose.
 Yeah, I think that would be, I don't know, like the future of programming looks more
 and more something I don't recognize from that, I think is very exciting.
 Is there some sort of, you know, between these two, is there some sort of adversarial setup
 that I could do?
 You have various models, like you have a model that generates new test cases, but at various
 stages, right?
 So for the clustering, you simply need to execute and observe the same outputs.
 Because I'm going to guess a model that makes new test cases doesn't necessarily make correct
 test cases.
 But is there also a model that makes test cases just sort of generates them, let's say,
 in a language model way, in a, you know, most likelihood way?
 Do you ever think of some kind of adversarial setup, given that DeepMind is a lot of in
 the space of like self play and sort of this reinforcement learning setting?
 Is there opportunities here for sort of systems to challenge each other to get better?
 Yeah, that's, it's very funny that you mentioned that because the project started off right
 after the AlphaStar project, basically.
 And so we had our minds were full of these types of ideas.
 Right.
 And so that's something that I've actually been very keen on since the inception of the
 project more than two years ago, to bring some notions of self play, curriculum learning,
 etc.
 I think that that would be very exciting.
 Unfortunately, generating new problems is an extremely difficult task, because first
 of all, your problems need to make sense.
 They need to actually be solvable.
 Right.
 So I can definitely see a world where we have many, many problems.
 And either they're way too difficult or they're nonsensical.
 And the other thing is we also have to come up with unit tests that work with the description
 of the problem.
 Right.
 And we have we have a data set of 12 to 13,000 problems, if I remember correctly, which is
 probably not enough for us to train a really good generative model to ask problems.
 So we haven't, we haven't really tried up until now.
 So I guess maybe I think one distinction I think is relevant there is that in AlphaStar
 and in a couple of other self play setups, they are symmetric.
 So you kind of expect the both sides to be improving all the time.
 Whereas in our case, it's less obvious how you might improve the problem maker over time.
 Maybe there is a I have no clue how these problems are actually made because humans
 need to make these programs.
 Right.
 If I look at a problem problem description like this, I'm like, this is this is insane.
 Not only is it very thorough, right.
 Also I have to somehow make sure that I as a maker of the problem don't make a mistake.
 And when I generate test cases, usually, you know, the example inputs right here are kind
 of small, but then I need to test like all the edge cases, right, to make sure that people
 have the correct algorithm, which means some are going to be very long and so on.
 So I almost have to write like a generator for, you know, these these long things.
 Maybe there isn't maybe there's a way to replicate that process of like how humans come up with
 these problems as because they're going to have like strategies and whatnot.
 They just they don't just sit there and go like, well, backspace.
 Right.
 I don't know, have you looked into do you know how these problems are made, like on
 a mechanical level?
 So I think we've been focusing a lot on the solving aspect of things and a lot less than
 the generating problems aspect of things.
 I have I have a healthy respect for the difficulty to generate problems that people can actually
 solve.
 Right.
 So I think we've been doing exams and thinking this is no fun.
 And then I know a lot of people who are teachers who have to actually devise exams.
 I think, wow, this is even less fun, actually.
 But yeah, I don't think we have a really good grasp on the human generative process for
 this thing.
 It would be really interesting to discuss with problem makers to see what are the strategies
 and whether or not we can try to replicate that and when possible direction would be
 to actually help them.
 That would be quite cool.
 Yeah, I think that's sorry.
 I think that's a great idea, actually.
 Like I I'm really quite interested to go and ask them myself now, I think.
 Maybe like if I had to do I would look in a computer science textbook and for like algorithms
 and then dress them up in some kind of story.
 That seems to be like what what a lot of problems are.
 But yeah, in terms of doing it mechanically, maybe that would be even harder than generating
 the solutions because like lots of people upload their solutions to GitHub.
 But I guess I expect there would be less data on how to create problems on.
 Yeah.
 Yeah, I was I was exactly I was more thinking of there must be some process because also
 these these people have to come up with new and new problems, right.
 And there's only so many algorithms and something like this backspace problem.
 It's very intricate, right?
 There is not really like an algorithm that I can just poof apply like I really have to
 think through stuff.
 One of my questions is that you hear the test cases, the public test cases, they're kind
 of samples, right?
 For you also to think through as a human.
 But very often, the testers, they also want to test not only whether you have the correct
 algorithm, but also whether you have the sort of correct runtime algorithm.
 Because you know, I can write an algorithm, you know, in I don't know, like if I have
 an O of n squared, that might not be the algorithm the tester is looking for.
 So they want like the O n log n.
 I'm having trouble writing the O n log n algorithm, right?
 Because one is really easy to implement.
 And one is actually the challenging one.
 So they will make deliberately like very large hidden test cases, so that my my naive algorithm
 would either go out of memory or out of time on the evaluation server.
 And this is something that you would not capture with just filtering on the public test cases
 as as your algorithm does.
 Your algorithm would think, well, I've solved the problem, right?
 I've come up with a solution.
 The naive solution will probably even be the more likely one given the language model.
 And then right and then it's it's filtering, it's clustering is like, well, all of this
 seems just fine, right?
 How do you have any grasp on how good you are on these types of problems?
 And is your model does it have some strategy to overcome that?
 Yeah, I think I can take that.
 The main answer here is that we just don't we just don't do it.
 We when we actually like looking at what our real self rate is, we had to do a lot of manual
 checking of solutions to check that they were meeting asymptotic complexity requirements
 of that we expected the problem to actually have.
 I think you do you mention before the call or in your question about clustering to buckets
 by by time or memory, I think you wrote that down.
 Did you have this in the paper or was this something I came up with?
 I don't I don't think that you came up with.
 Okay, yeah.
 Yeah, is this I mean, is this is this viable or is this like a bad idea?
 Or?
 Yeah, I guess I just had a thought on that.
 I think it's quite a cool idea.
 Maybe that particular implementation of looking at time and memory usage of of inputs like
 definitely is in the theme of, you know, executing the program and saying what happens.
 So I think an idea along that lines is is actually worth a go.
 One thing I would say is that a lot of these problems, I think, when you write the solution,
 which is asymptotically better, usually has like a big constant factor in front of it
 or a constant additive complexity.
 So you'd have to kind of consider that and whether that is going to adversely affect
 which solutions you're removing, maybe you're removing the thing which actually is going
 to have actually the asymptotic complexity.
 I think we could probably use it to cluster, right?
 Because then we had different if you had the same different asymptotic implementation,
 you would have different different values.
 But choosing directly according to like trying to rank them, depending on the performance
 on very, very small unit tests, we would probably I mean, my intuition.
 And our intuition, I guess, is is that we'd have to be extremely careful how we do that
 and not to overfit too much to that particular metric.
 So something that I want to point out, though, is that, yes, sometimes we have what we call
 slow positives, which are correct, except that they're impractical.
 But still, I already find that to be quite impressive, because some of these problems
 we go for the naive approach, but it's not completely evident that the naive approach
 would even work.
 So there's this thing like you want to remember, coding mentor told me about just make it run,
 make it right, make it fast.
 So we make it run, we make it right.
 Now all we have to do is to make it fast, which admittedly is a really difficult problem.
 I think I wouldn't be too worried that the clustering might not work.
 I would be more worried that the language model itself might not even, you know, might
 just jump on the sort of more likely naive implementation and never actually get to output
 the very different, possibly more efficient implementation, because these two things,
 they don't often look similar.
 They often look very, very different from each other.
 And yes.
 I think another issue is in our pre training sets on GitHub open source code, probably
 very, very fast, efficient programming isn't the majority of what's on there.
 So it might be that there's a bias towards simpler, more naive solutions already when
 we start fine tuning.
 So of course, we'd have to fight against that.
 With respect to the sampling and whether or not you can output something, you have a lot
 of tricks to increase your sampling diversity.
 One of the most notable things is that you have this prefix right here, which I found
 quite quite genius.
 I think in general, the approach of including sort of unknown things like that you would
 only know at training time, like things about your labels into the prompts, and then having
 that as sort of like a dial where you can control the model.
 I think that is a very cool, very cool idea.
 And I think you've shown quite quite impressively how that can help.
 You use it mostly to use it to to vary the outputs of your model.
 But that brings me like, given that we have to do all of these things to increase diversity,
 do you think maybe where our sampling procedure as such isn't a very good one?
 Because we have to do all these tricks, like could we fundamentally remake our language
 models or our generative models to to be more like diverse, let's say?
 Yeah, so I do think you're right.
 And we're not equipped with the right tools just yet.
 Right now we have this very crude setting to tune, which is a sampling temperature.
 But this means that we have very little control over how qualitatively diverse our samples
 are going to be.
 All right, so we're searching over the model distribution in an extremely crude way, which
 is basically pointing it into a general direction and say, OK, try to take as many sample ports
 as you can in that particular direction.
 But it seems important to me that we should be able to branch out in different directions
 only at fairly select decision points, not on every step.
 And we don't have a proper mechanism to do that.
 So we have high hopes for top K and nuclear sampling or for our sampling being guided
 by a value.
 But as we report in paper, this didn't really bring significant improvements.
 And I think another thing here is that we are sampling very independently.
 We're not taking past samples into account.
 When sampling a bit more autoregressively at the level of samples could probably be
 an interesting thing to explore.
 Yeah, I had one other point there.
 Since we sample from the models autoregressively, maybe this isn't really related to the diversity
 point, but to something in general, that's clearly not how I do things at all when I'm
 writing code.
 I usually write something, I write a sketch, and then I iterate over it in random bits
 of the code.
 So it's possible that that also is something that needs to fundamentally change by the
 way that we sample from models.
 I haven't looked much at the outputs the model generates, which astounded me.
 Just seeing this and seeing it output from a language model is astounding by itself.
 But also, it's very instructive.
 On the right, you even do a little bit of analysis and say, you know, these lines are
 this, these lines are this, these lines are this.
 Did you generally find that throughout your solutions?
 I haven't looked at many more solutions, to be honest.
 Did you generally find that code is interpretable, you know, very, very sort of instructive?
 Or is this a particular problem that you've picked out and to show kind of like, oh, look,
 the model solves the problem in an understandable way?
 Or did you, was most of the output cryptic or understandable?
 Yes, I think I looked at a fair few, you know, individual solutions when I was doing the
 analysis for this paper.
 I think in general, so actually, to be clear, like we did definitely pick this example as
 something that, you know, illustrates what's going on.
 But in general, you know, the model does produce things which you can read and understand what's
 going on.
 I think you have to, you know, and that's kind of expected in a way because we're training
 on human data, right?
 We're training to mimic the way that human programs look.
 So that's not crazy.
 But when we fine tune, competitive programmers write very unreadable code.
 So that's another thing to bear in mind.
 They will use a lot of type devs in C++, for example, a lot of crazy helper functions.
 And that's also something you see a lot in some of the solutions.
 You'll see these like huge copy pastes of code which like passes an input in an efficient
 way.
 A lot of that is dead code and it doesn't actually get used.
 And that's consistent with some of the competitive programming, like real solutions.
 But yeah, I guess like in this, you know, maybe it's because we filter for public tests
 as well, like in particular, the solutions which are correct seem to be fairly interpretable
 and make sense.
 But yeah, on rare occasions, like the implementation is quite difficult to understand.
 But yeah, I think if you want to look into that a bit more, we do have the tool, alphacode.dmin.com,
 which Remy and Julian worked on.
 And there's also some commentary on there, I think, from Petr, who works at Google, about
 what the model is doing.
 And I think in the samples he looked at, generally, he was quite happy that a lot of them seem
 to be doing something that you would expect in a reasonable way.
 I mean, it's distantly possible that you write something that just passes all the test cases
 but isn't actually correct.
 Like we're sampling so many things, like this might be not very likely.
 So it's definitely possible.
 And we did a fair amount of work actually generating new tests to try to make sure that
 that didn't happen.
 I remember somewhere, maybe a little bit under a year ago, we took a deep dive on our solved
 rate and we were trying to figure out whether it was the actual thing or whether actually
 we were gaming the problems.
 And we realized that there was a significant percentage of our solutions, quote unquote,
 which were getting the system.
 And the possible reasons for that were that actually there was very little coverage because
 there were many tests, but the answer was always the same.
 Sometimes you have yes, no type of things.
 And you look at the private test and the answer is always yes on the 40 private tests.
 And so the model will try, if you sample from it a million times, it will try to just print
 yes.
 That's probably going to happen.
 And for other things, we just had very, very few tests.
 So we filter out the problems, we had too few tests, but we also mutated the tests to
 add new ones to make sure that this didn't happen.
 And I think we went down from, I don't remember if it was 40% or maybe even 60% false positive
 rates to about 4% in our final data set, which is still significant, but we've found that
 was a reasonable and acceptable amount of false positives.
 I don't think I mentioned this in the video too much, but you have this kind of fuzzing
 approach to generating new test cases where during training, you know the correct solutions.
 So you can essentially generate new correct test cases by using the correct solutions
 that you know are correct, which I found, yeah, it makes sense.
 I think in this space of programming, you can do a lot of these things, which is neat.
 So what happens basically is we mutate programmatically the inputs of the tests that we already have,
 and then we run the human correct solutions on them.
 And then if we filter these new mutations, because some of them might not actually be
 correct inputs, and we figure out whether the human solutions actually agree on an output.
 And when we have a sufficient level of agreement on a given output, then we add this mutated
 input to the output that's generally agreed upon.
 Now, you mentioned before that you had high points and low points during the process of
 this project.
 Again, I can imagine that might be one of the lower points when you realize, wait a
 minute, all we do is false positives.
 Could you, I don't know, could you let us in maybe on what was sort of the lowest point?
 Was there a moment where you thought, ah, this isn't going to work out, you know, after
 all this time?
 And what did you do to overcome these things?
 That's a tough question.
 When was I think the lowest point probably wasn't the same for all the members of the
 team, right?
 I think we did, because we were working on slightly different ideas most of the time.
 But I think there was in the middle of a project, there was basically a month where we had very,
 very little progress.
 And so we had these meetings every week when we would see what was the best performing
 thing and it was still the same thing.
 So there's that, that was definitely no point for us.
 And maybe like also when some of the big ideas that we thought were going to help didn't
 pan out.
 Like for instance, when we realized that for whatever reason, it was just too hard to train
 a really good value function and we weren't going to be able to leverage all of the methods
 that this would have unlocked, which we did rely upon at least initially in our main map.
 So yeah, that would be my answer.
 I definitely had a couple of those myself.
 But I think in general, a lot of the times we realized that we got results which weren't
 actually true because they were false positives.
 Later on, we did claw back a lot of the gain.
 But I think that's just maybe the scientific method at work.
 We kind of proved us, we tried something and then we realized actually it wasn't working.
 But yeah, I think having our metric to guide us there really helped us get through those.
 I think we were well served by a somewhat skeptical approach when we had a result that
 looked good to be true.
 Our initial thought was okay, this is good to be true.
 Where's the issue?
 And more often than not, there was actually a bug that we found.
 Once you released the, let's say the paper and so on, I think a lot of comments started
 coming in.
 Did you have a criticism that, what is the most valid criticism that you've encountered
 that you didn't foresee?
 Obviously, you have a lot of limitations at the end of the paper and you make it very
 clear like this is one niche, this is this, there's limitations here.
 Is there something that people brought up and you were like, oh yeah, I didn't think
 of that.
 That's a good point.
 There's a few things, it's a difficult question generally, but there's a few things definitely.
 Generally, as we said, we've been very happy with how the work was received and we've gotten
 a lot of constructive feedback.
 Dima Badanoff's Twitter thread is a good example, for instance, where he outlined why he thinks
 and we do agree with him that we're still a long way from top level human performance
 on this task.
 I was also made aware that the data that we put on alphacode.deepmind.com was actually
 not correct.
 I had filtered the correct solutions wrong.
 So again, underlining the importance of doing that right.
 So I thank everybody who told us, well, I don't understand this correct solution.
 It's actually not correct.
 And they were right.
 So now we've fixed that.
 So if you go to alphacode.deepmind.com, you will get actually correct solutions.
 And then something that surprised us, but I don't know whether it's valid or not, is
 that a fair amount of people seem to think that the average human competitor on codeforces.com
 is not very good, which I think we have a fairly different view.
 So I'm not sure I would say it's valid, but it was certainly surprising to us.
 And then in terms of the limitations of the model, we thought a lot and just a bit of
 what we thought were the weaknesses.
 So I'm not sure that I've seen anything that we hadn't already identified.
 Cool.
 Where do you see this more in the real world?
 We talked about programming, competitive programming, maybe a future where I can just write a bunch
 of unit tests and this will go fine.
 But there are obviously applications beyond this.
 Are there people maybe in your team that are already eyeing or maybe you have some ideas
 of this?
 Where could this be used outside of programming?
 Just the techniques in here and the methodologies.
 Do you see some sort of semi-obvious transfer to a real world problem other than coding?
 I think generally speaking, there's going to be a lot of downstream applications for
 general purpose problem solving AIs.
 To our team, we've been thinking a lot about programming and less about non-programming
 applications.
 So I think Farfakir, there's some natural directions, which include developing tools
 to make coding easier, as we already touched upon with automated test generation, smart
 autocomplete, etc.
 Or maybe tools to make it easier to learn how to code.
 So you could imagine an AI that can comment and suggest some improvements to your code,
 etc.
 But I think the applications that could be used to democratize programming are definitely
 on our radar.
 In terms of applications not directly related to programming, I haven't thought too much
 about that.
 I'm fairly certain that problem solving is sufficient in general so that we will find
 interesting applications, but we haven't been too much on the lookout for that.
 I think you're right to point out a couple of those ideas, Yannick.
 And I think Codex has also shown us that this works.
 You can build a product out of these kinds of models, and people are really happy with
 it.
 So it's definitely something that we're thinking about, but I think we definitely haven't concretely
 made any decisions at all or finished brainstorming even, whether that's something that we'd like
 to do.
 But yeah, I think maybe to go back to one thing that Remy mentioned earlier is that
 the methods that we use are actually pretty general, I find, as far as programming goes.
 The filtering, which is the really big one, could definitely be used in an application.
 But a lot of what softwrench does is just nothing to do with writing code.
 And one way I guess I would think about it is what we've done is take a description of
 a problem and actually a complete description of a problem and map that to code.
 But really, I find in my day-to-day, I'm spending maybe 50% or more of my time talking to people
 and writing that description, if that makes sense.
 Yeah, Alpha requirements engineer is the next paper.
 Is there anything else you want to get out about this paper?
 Can people somehow get started with or get into this type of research or anything you'd
 want to communicate?
 I think we'd be really excited for other researchers to work on this.
 I know some other researchers are already working on this problem, but our goal is that
 as many as possible actually work on this problem because any gain we make here is going
 to be distributed.
 So that would be really nice.
 And that's why we released our data set, which we spent a fair amount of time on and we think
 is a really good tool to approach these problems.
 As we showed in the paper, you don't need huge models to actually start solving problems.
 So you can do that with less resources.
 Of course, there's the issue of having to sample a whole lot, but I would say that's
 a very exciting research direction to actually reduce the amount of samples you have to take
 to solve these problems.
 Peter, any messages for anyone listening?
 I think as Remy said, the fact that we released the data set is clear that that's the main
 point that you should start.
 But I think in general, I'm optimistic not just about competitive programming, but about
 people working on programs in business in general with machine learning.
 So I can only encourage people to go and do it.
 And actually, I should say that as a programmer myself, I'm quite optimistic that working
 on this kind of problem is going to make my life a bit easier.
 In this case, Peter and Remy, thank you very much for being here.
 This was a lot of fun.
 I learned a lot.
 And I hope to see the alpha requirements engineer in the future.
 Thanks for having us.