On the Measure of Intelligence by François Chollet - Part 4: The ARC Challenge (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper chollet keras google francois intelligence iq iq test deep neural networks prior skill performance measurement measure test number intelligent smart learning generalization ability experience humans evolution nature nurture psychometrics range adaptability arc kaggle difficulty entropy core knowledge objectness navigation contact agent goal
In this part, we look at the ARC challenge as a proposed test of machine intelligence. The dataset features 1000 tasks that test rapid generalization based on human core knowledge priors, such as object-ness, symmetry, and navigation.

OUTLINE:
0:00 - Intro
0:55 - What is ARC?
6:30 - The Goals of ARC
10:40 - Assumed Priors & Examples
21:50 - An Imagined Solution
28:15 - Consequences of a Solution
31:00 - Weaknesses
31:25 - My Comments & Ideas

Paper: https://arxiv.org/abs/1911.01547
ARC: https://github.com/fchollet/ARC

Abstract:
To make deliberate progress towards more intelligent and more human-like artificial systems, we need to be following an appropriate feedback signal: we need to be able to define and evaluate intelligence in a way that enables comparisons between two systems, as well as comparisons with humans. Over the past hundred years, there has been an abundance of attempts to define and measure intelligence, across both the fields of psychology and AI. We summarize and critically assess these definitions and evaluation approaches, while making apparent the two historical conceptions of intelligence that have implicitly guided them. We note that in practice, the contemporary AI community still gravitates towards benchmarking intelligence by comparing the skill exhibited by AIs and humans at specific tasks such as board games and video games. We argue that solely measuring skill at any given task falls short of measuring intelligence, because skill is heavily modulated by prior knowledge and experience: unlimited priors or unlimited training data allow experimenters to "buy" arbitrary levels of skills for a system, in a way that masks the system's own generalization power. We then articulate a new formal definition of intelligence based on Algorithmic Information Theory, describing intelligence as skill-acquisition efficiency and highlighting the concepts of scope, generalization difficulty, priors, and experience. Using this definition, we propose a set of guidelines for what a general AI benchmark should look like. Finally, we present a benchmark closely following these guidelines, the Abstraction and Reasoning Corpus (ARC), built upon an explicit set of priors designed to be as close as possible to innate human priors. We argue that ARC can be used to measure a human-like form of general fluid intelligence and that it enables fair general intelligence comparisons between AI systems and humans.

Authors: François Chollet

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there and welcome to the last part of On the Measure of Intelligence by François Chollet. This last part concerns the ARC challenge that Chollet has proposed, or the ARC dataset, which stands for the Abstraction and Reasoning Corpus. And we're just quickly going over the dataset, look how it's built and discuss what kind of solutions might be relevant right here. So if you haven't seen the last videos in this series, this is the last one of a series, you might not exactly know what's going on. But I think you can you can keep up pretty well because this part is fairly independent of the other parts. And it's just cool to think about even if you haven't seen the other ones, I encourage you to go see the other ones, but it's not necessary. Okay, let's jump in. So the ARC is a challenge currently running a Kaggle challenge. But in essence, it is a dataset. And let me just jump into one of the tasks of the dataset. So in this dataset, you always have the task in the following form. So you always have multiple input examples like this or say these are called the training examples. And then you have a test example. In this case, you have three training example one test example. So an entire if you think of this in a machine learning way, this entire thing here is your x and this thing here is your y. Okay, so the label is going to be the output of the last example that you you don't know that now in the course in the training data set you do, but in the test you don't. So each one of these, as I said is is is demonstrated. These are the demonstration examples. And then you're supposed to sort of learn the regularity out of the demonstration examples. And then on this test example, you are supposed to apply this regularity that you learned. So in here, a human can fairly accurately see that there are these black squares in each image, and that in the training samples, the output will always sort of exactly match into the place of these black squares. As you can see, this is like a high rectangle. It goes here, it has the same amount of tiles and so on. And you can also see that whatever colors are in here sort of are the continuation of a symmetric pattern. So here, this is exactly the same as up here, but you know, flipped or turned by 180 degrees. So there is a notion of symmetry right here. So technically, one could compute this one would say, oh, that's probably going to be the three rows and this bunch of things. And it's probably going to be the same as this one down here, but just flipped on its head. So as a human, you get this even without a description, you realize like, oh, this is like a regular pattern, it's symmetric, there's a hole in it. And apparently, the thing here always fills the hole, I can see that, you know, three examples are enough for me to confirm that that's what's going on. And I see the hole here. So I'm going to do the same thing. So you can already see how these things are constructed in every, this is not the only task, by the way, this is just one task. Okay, there are 1000 tasks in this data set of this sort of nature. Now they're not always three demonstration examples, I believe there can be more or less. But what's always the case is they always each of these training examples consist of these demonstration examples and these test example. Each of the demonstration examples consists of an input grid and an output grid, the input grid and output grid, they can be anywhere from one by one to 30 by 30. Okay, anywhere in between that. And the colors here, I believe there are nine different colors that can go, they're just encoded by nine different numbers. But there are nine different colors that these things can have, you can see black, blue, orange, red, dark blue, and so on. And the output grid exactly the same. Now in this test example, you can only see the input grid, you cannot see the output grid. And that means you don't even know how large it should be. You can see right here, they're not all the same size, the output grids. In fact, not even the input grids have to be always the same size. But you have to now come up with an output grid. You have to first decide how big it is. And we've here we've determined since the whole has three rows, we're probably going to make three rows and it has like seven columns, we're probably going to make seven columns. And that's the sort of thing you have to do. And then not only do you have to decide how big it is, you now have to decide in each cell what color you put in. And only if this thing exactly matches the train or the test label, you get a you get a point. Otherwise you get no point. So in the training task, there are I believe 400 of these tasks and then there are 400 more as test split, but these are still public and then there are 200 that are secret. There are I guess, part of this Kaggle challenge. Yes, the training set features 400 tasks, while the evaluation set features 600 tasks. The evaluation set is further split into a public evaluation set of 400 tasks and a private evaluation set of 200 tasks. All tasks are unique. And the set of tasks and the set of training tasks are disjoint. Sorry, of test tasks and training tasks. The task data is available at this as you can see right here. So I really hope that Cholay will keep these 200 tasks as a secret, even after the Kaggle challenge, because it's going to be fun for people that might want to get into this later. So here are the goals of this data set. They want to stay close to these psychometric intelligence tests. They say in particular, it should be solvable by humans without any specific practice or training and probably also without any language instructions. So you just be able to set a human in front of it and the human should be able to solve it or a large portion of humans should be able to solve it. Ideally, this test would also differentiate humans from each other. But at this point, we want to simply assess machines. So they say focus on measuring developer aware generalization rather than task specific skill by only featuring novel tasks in the evaluation set. And the novel tasks are unknown to the developer of a test taker. So if I develop a system, I don't know what are these 200 tasks that Cholay keeps hidden. I simply submit my code and I'll figure out if my code does well on them. So they say they want to feature highly abstract tasks must be understood by a test taker using very few examples. That's what you saw. You don't have a big training example to learn that this task is about symmetry and hole filling. You only have three. And from three, you need to recognize what's going on and produce the output of the test sample. Quality of control for experience by only providing a fixed amount of training data for each task. That's what we saw. And only featuring tasks that do not lend themselves well to artificially generating new data. So it's not like ImageNet where you can go on the internet and find a whole bunch of images or some NLP tasks where people pre train on all of Wikipedia and all of the books in the world because they want to understand language better. These tasks are supposed to be such that it makes no sense for you to go out and try to find more data or find similar data or pre train your model on something. And then lastly, and this refers to the last few chapters we looked at explicitly describe the complete set of priors that it assumes and enable a fair general intelligence comparison between human and machines by only requiring priors to those innate human close to innate human prior knowledge. So that means that whatever human have whatever humans have as a prior built into them by let's say evolution or that most humans have picked up through life. Those are the things that you have to explicitly point out. So and you require that and you have to point them out. Sorry, explicitly describe them such that I as a developer of a system can build them into my system such that it's a fair comparison. In the last chapters, we looked at the fact that a fair intelligence comparison is only fair if two systems that are compared to each other have the same amount of experience. And here we control that by only providing a fixed amount of training data and also have the same prior knowledge. And here we simply do that by listing the human priors that are required for the tasks that we think that humans have. And then we enable the developers to explicitly build those into machines. So I would maybe build a little calculator module into my AI that solves this tasks. Okay, so they say he each task consists of a small number of demonstration examples, 3.3 on average, and a small number of test examples, generally one, although it might be two or three in rare cases. Each example consists of an input grid and an output grid. Each grid is a literal grid of symbols. Each symbol is visualized by color. There are 10 unique symbols, a grid can be any height or width between one by one and 30 by 30, so it doesn't even need to be square, right? And as I said, you need to provide your own output grid as an AI taking this test. So here are the priors that this test assumes. And we're going to look at some examples that make it explicit like some tasks in the training set that where you can see these priors in action. There's an object nest prior where the task assumes that the AI or the task tests that the AI understands something about objects. So these are tasks that you can only reasonably solve if you know something about objects, like you would, a human would recognize or would, you know, would recognize that these things might represent different objects, right? Now that's mainly, I think also due to the black background helps, but you would even recognize this with another background or here, the different colors indicate that those are two different things, even though those two pixels here touch and are different from black, you would recognize that those are two different things because they have different color. But you would generally recognize one of these things as an individual object. If you're not given anything here, you see, for example, a denoising task as a human, you can pretty quickly see what the task is about, right? There appear to be these green things. They're all rectangles and there appear to be these blue things. And on the right side, there are no more blue things. But the now it's not always that when there was a blue thing, there is now a green thing only here where it was sort of inside a green thing is now a green pixel. Whenever there was a blue pixel outside in this black area, then there is now black. So this is sort of like the blue things were noise and you're able to remove it. This already tests a lot of assumptions. A lot of these priors, a lot of understanding of the world. So there are objects, right? As human understands that objects are square in this case, or rectangles, the human understands that it that we need to remove the blue things going over. And the human understands that somehow this inside relation, right, if something is inside or outside of one of these rectangles, and that determines whether we have to turn the pixel green or black. You can, I mean, think about how you would train a machine to do something like this. It's not easy, especially if you don't know that this task is coming. Imagine for all of these things, you don't know that the task is coming. This is just one of 400 tasks that you know of. There are 600 tasks that you don't know of that are similar, but also in a way completely different. Here's another tasks that object influence via contact. So this is your first demonstration example. A human pretty quickly recognizes there appears to be red thing and a blue thing, and then they appear to be together. And then in the next thing, you see, oh, there appears to be a blue thing and the red thing in the next thing, they appear to be together. And if you look here, it always appears to be the red thing going to the blue thing in the most direct way. So in the in the along the grid. That's all that the human needs to see two examples and the human most humans will already make that inference and can now solve if there is like, if there now is a test example, where the blue thing is like, the blue thing is down here. And the red thing is here like this. And it asks you what comes next, you know, you know that the red thing is going down to the blue thing. But it's very hard to train a machine to do this. So I like this test, because it's sort of a different test. And I believe the test these tests weren't procedurally generated. These tests were actually generated by sholey or, you know, by by actual humans. That's pretty cool. And 1000 tasks like this is going to be very hard to solve. There are even more abstract priors like goal directedness. So now you here you can already see this a little bit in that you can say, well, the red thing wants to go to the blue thing. So there is a notion of time involved, maybe. There's also counting and numbers and numbers prior. So here you see like a time process. So in this demonstration example, you see blue things here, red, big thing. And then the next the output grid is this green thing. And as a human immediately recognize, okay, so it shoots out from the blue thing, the green thing shoots from the blue thing, hits the red wall and goes here. Try to make a machine understand this. This is insane, right? So if you look at the more examples, it all it appears that the blue thing always comes from somewhere like the side of the image, and the green thing comes out obviously from whatever is not at the at the border of the image and then bounces off the red thing if it hits the red thing. Now here you can you can already see what's going to happen. Remember your AI would need to first determine aha, okay, all of these output grids, they seem to be the same as the input grid. So it would need to explicitly construct the output grid in the same manner as the input grid because it understands this right? This is not the same in every task, then it needs to recognize the red thing that stays in every one. So it needs to put the red thing here right from from here. And then it needs to recognize the blue thing stays as well. And then most most shockingly needs to recognize, okay, I will draw a line in pixels and lines in pixels are hard here. And then as soon as it would hit the red thing, it bounces off into the other direction. So from just these three examples, the machine has to understand that and correctly output the exact solution, not an approximate solution, the exact solution. Okay, so yeah, there are these basic geometry and topology priors like lines, rectangular shapes, symmetries, rotations, translations, shape upscaling, containing being contained, drawing lines, connecting points, and so on. Now, let's look at some more examples. These are fun, right? Check out this one here. So you see green, red, and then somehow the green connected to the red. So this is an example of that has many of these priors in many of these concepts in there is goal directedness, you can already sort of form the hypothesis that the green wants to go to the red. But also you see that somehow it sort of appears to the blue things seem to be maybe obstacles and it appears to change direction when it encounters an obstacle like here. So here, you see the example, and you probably confirm so your hypothesis could be it always goes until it hits and then it changes direction towards the red thing, right? Always towards red thing, because it's not always towards the right because you return toward the left. So it goes somehow towards the red thing and so it's pretty ambiguous in this situation, but you can also make the assumption that if it's ambiguous, it goes towards the middle, maybe, maybe. So here, again, now we're actually confirming probably so we go towards the red thing, which would be towards this direction, then we hit an object, then we go towards the red thing until we hit an object, and then we go here. Also see that these grids here are not the same size. So it's not always the case that the grids of within the same tasks are even the same size. So now here, again, your AI would need to recognize what size of grid it needs to draw and what the result is. So it would need to copy this entire grid and also change these pixels right here to be green pixels. That's hard. I mean, that's I find I find this to be pretty hard. This is the line extrapolation and turning on obstacle and efficiently reaching a goal prior. That's crazy. And is there more? Yes, there is two more, I believe. Yeah, those are the last examples. So in this one, you can see right here, there appear to be objects, which there's this blue objects appear to be the same and there are these red, and then the output grid is one of these blue objects. Okay, so here we again see different objects, the output grid is one of them. So as a human, you can already recognize the output grid is probably always going to be one of these objects. And now we need to decide on which one. So we can formulate the hypothesis that it's probably going to be the one that's the most like here, there's three of the blue ones here, there's four of the yellow ones, that's more than any other. And this here confirms our hypothesis that the it's the object that appears most often. Now again, see that there is this notion of object ness. You need to upscale somehow. No, this is not upscale because the grid is the same size. It's simply the image that's upscale. But you need to somehow focus be able to focus in on one of these objects, I need to count them, you need to compare the counts via each other. And now here you can pretty easily see that the output grid is going to contain one of those blue things as a human. And here, it's it's sort of a symmetry filling task. Now as a human, you need one demonstration to get this. Maybe you need more, but many tasks involve some sort of symmetry. Okay, drawing the symmetrized version around the version of a shape around a marker, that's going to be fairly hard for a machine to learn without without the developer knowing that this task is coming. Okay, they highlight some differentiations to standard psychometric tests. But what I find interesting here is that this thing, what a solution to arc may look like and what it would imply for AI applications. They say we have found art to be fully solvable by humans. So they set a human in front of every, every one of these tasks, and it's solvable. While many arc tasks are intellectually challenging human test takes us appear to be able to solve the majority of tasks on their first try without any practice or verbal explanations. In effect, in this task, you get three tries at each at each of the at each of the problems, you get three, three tries and the humans can already solve it in one. So that just shows you shows you how cool humans are. So here is a shawley suggests a solution approach says by start by developing a domain specific language capable of expressing all possible situations, all possible solution programs for any arc task. Since the exact set of arc tax is purposely not formally definable, this may be challenging. The space of tasks is defined as anything expressible in terms of arc pairs that would only involve core knowledge. So core knowledge is this set of human priors that we discussed last time like objectness and symmetries and geometric shapes and navigation and so on. So he asks you to basically develop a DSL that can capture all the different tasks. So so kept basically define a formalism of these tasks. But it's hard because you don't know what the tasks are going to be. So your best bet is probably to make a formalism that completely over represents what the tasks can be. It would require hard coding the core knowledge priors from 3.1.2 in a sufficiently abstract and combinable program form to serve as a basis functions for a kind of human like reasoning DSL. We believe that solving this specific subproblem is critical to a to general AI progress. Basically says whenever we can describe this is like saying that this AI progress will make a big step once we can formally describe human priors. And while true this I feel the hardness of this problem is as hard as actually building general artificial intelligence or very close to it. So it is a bit of a like how to how to go how to build a GI step one build a GI that's sort of and not exactly but it's kind of what this says. Right. So if I could actually have this DSL to describe every single task and I could do it you know such that it is not not super over capturing all the tasks then I would be able and I would have described human core knowledge in a sufficiently accurate degree that I could just you know build a GI. So he goes on says given a task use the DSL to generate a set of candidate programs that turn the input grids into the corresponding output grids. This step would reuse and recombine sub programs that previously proved useful in other arc tasks. So says whenever you have captured the core knowledge or whenever you have captured the problem space in a formal language you can simply use that formal language to express whatever your input is so the that turn the input grids into the corresponding output grids. So you would put in these demonstration examples and describe this with your formal language that you have and you can somehow reuse and recombine sub programs that previously proved useful for basically asking you to write to come up with source code that would generate these demonstration examples in the language of your DSL. And then he says select top candidates among these programs so you would generate multiple versions of source code that generate these things based on a criterion such as a program simplicity or program likelihood. Note that we do not expect that merely selecting the simplest possible program that works on training pairs will generalize well to test pairs. And then use the top three candidates to generate output grids for the test examples. So I hope the approach here I feel it makes sense but it is sort of over hopeful in my mind and that's mainly because of step one. So step one asks you to come up with like a programming language that can capture all the tasks in this all the tasks in the data set even though you don't know what the tasks are and that has this human core knowledge in inside of it in a in a formally describable way. And then once you have that programming language you would if you're given this task where you have you know a bunch of these demonstration you have a bunch of these demonstration things and then you have the test thing you would generate all the programs that would produce these demonstration examples or that would given the demo given the input grade would produce the output grid right you would generate all the programs and then you would select somehow among all these programs the one that you think generalizes the most and you would use that program to put this in and get out the solution. They say it's probably it's not always the simplest program not always the shortest program maybe who knows like I feel step one is the kind of the crucial issue here. Okay so they say they make some claims here and about what this what this would bring the community we posit that the existence of human level arc solver would represent the ability to program and AI from demonstrations alone only requiring a handful of demonstrations to specify complex tasks to do a wide range of human relatable tasks of a kind that would normally require human level human like fluid intelligence. As supporting evidence we note that human performance on psychometric intelligent test which are similar to our is predictive of success across all human cognitive tasks. Further we posit that since an arc solver and human intelligence would be both founded on the same knowledge priors the scope of application of an arc solver would be closer to that of human cognition making it such a solver both practically valuable and easy to interact with and would produce behavior that is in line with human expectations. Okay so they're making the same argument that anyone before has made but they condition it on some things and this is I think the conclusion of the entire article here of on the measure of intelligence because people had this hope and they say that here claims are highly speculative and may prove incorrect much like Newell's 1973 hopes that progress on chess playing would translate into meaningful progress and achieving a broad range of cognitive abilities especially if arc turns out to feature unforeseen vulnerabilities to unintelligent shortcuts. This is the AI effect and basically means that whenever you think a task the solving of a task represents AI and then you actually see the solution then the solution turns out to be not AI in the eyes of the human. So the human at first they would say oh this task really requires intelligence and then someone solves the task and they'll say oh that's not intelligence you kind of hacked your way to that and the expectation is that in this arc challenge there might be a hacky way to that but I mean the good question is when at what is there even a task like this arc challenge here could that is there even a possibility of a task where you wouldn't say that and I'm not so sure about this they seem to be more hopeful than I am but at least they say the arc challenge is founded on the same priors as a human has it gives you the same amount of experience as a human has right and therefore it is much more comparable to human intelligence. They go over some weaknesses right here of criticizing their own thing generalization is not quantified so they have a measure of generalization in the previous chapter but they don't use it right here test validity is not established data set size and diversity may be limited and so on but I in my mind this I would not consider this as like an AGI task or anything like this I'm pretty sure the solution to this will come in a form again where people don't really think it's it exhibits intelligence but I do like the task as such and as a machine learner I am very excited to think about how machine learning can go about solving this task and especially with what we've seen from something like GPT-3 it has exactly this kind of structure where you train on a giant data set blah blah blah you pre train your language model but then at inference time you input a bunch of these demonstration examples and you ask it for the next output so I feel that might be a good start for for doing it the question of course is what what then do you pre train this model on this GPT-3 for ARC what's the pre training data set for it and I guess that's going to be the challenge and probably going to require people to specifically program all of these priors into a data set generator for pre training so that would be my approach my approach would be write a data set generator for pre training and GPT-3 model to do these kind of tasks and in order to write the data set generator you'd have to basically program in all of these priors and that's not going to be easy because your best bet is to sort of put yourself into the shoes of Chalet and be like oh if I were to design a task what kind of things would I do and then try to capture that that's going to be your best bet your most honest bet with respect to the challenge is to try to as faithfully as possible implement something like an object Ness prior where cohesion and persistence are captured that would be the most scientifically sound approach to my approach alright so that was my take on the ARC data set if you have any comments I'm very excited to hear comments on this if you have already tried the ARC challenge have some insight I also welcome comments on that and with that I'll see you next time bye bye
 Hi there and welcome to the last part of On the Measure of Intelligence by François Chollet.
 This last part concerns the ARC challenge that Chollet has proposed, or the ARC dataset,
 which stands for the Abstraction and Reasoning Corpus.
 And we're just quickly going over the dataset, look how it's built and discuss what kind
 of solutions might be relevant right here.
 So if you haven't seen the last videos in this series, this is the last one of a series,
 you might not exactly know what's going on.
 But I think you can you can keep up pretty well because this part is fairly independent
 of the other parts.
 And it's just cool to think about even if you haven't seen the other ones, I encourage
 you to go see the other ones, but it's not necessary.
 Okay, let's jump in.
 So the ARC is a challenge currently running a Kaggle challenge.
 But in essence, it is a dataset.
 And let me just jump into one of the tasks of the dataset.
 So in this dataset, you always have the task in the following form.
 So you always have multiple input examples like this or say these are called the training
 examples.
 And then you have a test example.
 In this case, you have three training example one test example.
 So an entire if you think of this in a machine learning way, this entire thing here is your
 x and this thing here is your y.
 Okay, so the label is going to be the output of the last example that you you don't know
 that now in the course in the training data set you do, but in the test you don't.
 So each one of these, as I said is is is demonstrated.
 These are the demonstration examples.
 And then you're supposed to sort of learn the regularity out of the demonstration examples.
 And then on this test example, you are supposed to apply this regularity that you learned.
 So in here, a human can fairly accurately see that there are these black squares in
 each image, and that in the training samples, the output will always sort of exactly match
 into the place of these black squares.
 As you can see, this is like a high rectangle.
 It goes here, it has the same amount of tiles and so on.
 And you can also see that whatever colors are in here sort of are the continuation of
 a symmetric pattern.
 So here, this is exactly the same as up here, but you know, flipped or turned by 180 degrees.
 So there is a notion of symmetry right here.
 So technically, one could compute this one would say, oh, that's probably going to be
 the three rows and this bunch of things.
 And it's probably going to be the same as this one down here, but just flipped on its
 head.
 So as a human, you get this even without a description, you realize like, oh, this is
 like a regular pattern, it's symmetric, there's a hole in it.
 And apparently, the thing here always fills the hole, I can see that, you know, three
 examples are enough for me to confirm that that's what's going on.
 And I see the hole here.
 So I'm going to do the same thing.
 So you can already see how these things are constructed in every, this is not the only
 task, by the way, this is just one task.
 Okay, there are 1000 tasks in this data set of this sort of nature.
 Now they're not always three demonstration examples, I believe there can be more or less.
 But what's always the case is they always each of these training examples consist of
 these demonstration examples and these test example.
 Each of the demonstration examples consists of an input grid and an output grid, the input
 grid and output grid, they can be anywhere from one by one to 30 by 30.
 Okay, anywhere in between that.
 And the colors here, I believe there are nine different colors that can go, they're just
 encoded by nine different numbers.
 But there are nine different colors that these things can have, you can see black, blue,
 orange, red, dark blue, and so on.
 And the output grid exactly the same.
 Now in this test example, you can only see the input grid, you cannot see the output
 grid.
 And that means you don't even know how large it should be.
 You can see right here, they're not all the same size, the output grids.
 In fact, not even the input grids have to be always the same size.
 But you have to now come up with an output grid.
 You have to first decide how big it is.
 And we've here we've determined since the whole has three rows, we're probably going
 to make three rows and it has like seven columns, we're probably going to make seven columns.
 And that's the sort of thing you have to do.
 And then not only do you have to decide how big it is, you now have to decide in each
 cell what color you put in.
 And only if this thing exactly matches the train or the test label, you get a you get
 a point.
 Otherwise you get no point.
 So in the training task, there are I believe 400 of these tasks and then there are 400
 more as test split, but these are still public and then there are 200 that are secret.
 There are I guess, part of this Kaggle challenge.
 Yes, the training set features 400 tasks, while the evaluation set features 600 tasks.
 The evaluation set is further split into a public evaluation set of 400 tasks and a private
 evaluation set of 200 tasks.
 All tasks are unique.
 And the set of tasks and the set of training tasks are disjoint.
 Sorry, of test tasks and training tasks.
 The task data is available at this as you can see right here.
 So I really hope that Cholay will keep these 200 tasks as a secret, even after the Kaggle
 challenge, because it's going to be fun for people that might want to get into this later.
 So here are the goals of this data set.
 They want to stay close to these psychometric intelligence tests.
 They say in particular, it should be solvable by humans without any specific practice or
 training and probably also without any language instructions.
 So you just be able to set a human in front of it and the human should be able to solve
 it or a large portion of humans should be able to solve it.
 Ideally, this test would also differentiate humans from each other.
 But at this point, we want to simply assess machines.
 So they say focus on measuring developer aware generalization rather than task specific skill
 by only featuring novel tasks in the evaluation set.
 And the novel tasks are unknown to the developer of a test taker.
 So if I develop a system, I don't know what are these 200 tasks that Cholay keeps hidden.
 I simply submit my code and I'll figure out if my code does well on them.
 So they say they want to feature highly abstract tasks must be understood by a test taker using
 very few examples.
 That's what you saw.
 You don't have a big training example to learn that this task is about symmetry and hole
 filling.
 You only have three.
 And from three, you need to recognize what's going on and produce the output of the test
 sample.
 Quality of control for experience by only providing a fixed amount of training data
 for each task.
 That's what we saw.
 And only featuring tasks that do not lend themselves well to artificially generating new data.
 So it's not like ImageNet where you can go on the internet and find a whole bunch of
 images or some NLP tasks where people pre train on all of Wikipedia and all of the books
 in the world because they want to understand language better.
 These tasks are supposed to be such that it makes no sense for you to go out and try to
 find more data or find similar data or pre train your model on something.
 And then lastly, and this refers to the last few chapters we looked at explicitly describe
 the complete set of priors that it assumes and enable a fair general intelligence comparison
 between human and machines by only requiring priors to those innate human close to innate
 human prior knowledge.
 So that means that whatever human have whatever humans have as a prior built into them by
 let's say evolution or that most humans have picked up through life.
 Those are the things that you have to explicitly point out.
 So and you require that and you have to point them out.
 Sorry, explicitly describe them such that I as a developer of a system can build them
 into my system such that it's a fair comparison.
 In the last chapters, we looked at the fact that a fair intelligence comparison is only
 fair if two systems that are compared to each other have the same amount of experience.
 And here we control that by only providing a fixed amount of training data and also have
 the same prior knowledge.
 And here we simply do that by listing the human priors that are required for the tasks
 that we think that humans have.
 And then we enable the developers to explicitly build those into machines.
 So I would maybe build a little calculator module into my AI that solves this tasks.
 Okay, so they say he each task consists of a small number of demonstration examples,
 3.3 on average, and a small number of test examples, generally one, although it might
 be two or three in rare cases.
 Each example consists of an input grid and an output grid.
 Each grid is a literal grid of symbols.
 Each symbol is visualized by color.
 There are 10 unique symbols, a grid can be any height or width between one by one and
 30 by 30, so it doesn't even need to be square, right?
 And as I said, you need to provide your own output grid as an AI taking this test.
 So here are the priors that this test assumes.
 And we're going to look at some examples that make it explicit like some tasks in the training
 set that where you can see these priors in action.
 There's an object nest prior where the task assumes that the AI or the task tests that
 the AI understands something about objects.
 So these are tasks that you can only reasonably solve if you know something about objects,
 like you would, a human would recognize or would, you know, would recognize that these
 things might represent different objects, right?
 Now that's mainly, I think also due to the black background helps, but you would even
 recognize this with another background or here, the different colors indicate that those
 are two different things, even though those two pixels here touch and are different from
 black, you would recognize that those are two different things because they have different
 color.
 But you would generally recognize one of these things as an individual object.
 If you're not given anything here, you see, for example, a denoising task as a human,
 you can pretty quickly see what the task is about, right?
 There appear to be these green things.
 They're all rectangles and there appear to be these blue things.
 And on the right side, there are no more blue things.
 But the now it's not always that when there was a blue thing, there is now a green thing
 only here where it was sort of inside a green thing is now a green pixel.
 Whenever there was a blue pixel outside in this black area, then there is now black.
 So this is sort of like the blue things were noise and you're able to remove it.
 This already tests a lot of assumptions.
 A lot of these priors, a lot of understanding of the world.
 So there are objects, right?
 As human understands that objects are square in this case, or rectangles, the human understands
 that it that we need to remove the blue things going over.
 And the human understands that somehow this inside relation, right, if something is inside
 or outside of one of these rectangles, and that determines whether we have to turn the
 pixel green or black.
 You can, I mean, think about how you would train a machine to do something like this.
 It's not easy, especially if you don't know that this task is coming.
 Imagine for all of these things, you don't know that the task is coming.
 This is just one of 400 tasks that you know of.
 There are 600 tasks that you don't know of that are similar, but also in a way completely
 different.
 Here's another tasks that object influence via contact.
 So this is your first demonstration example.
 A human pretty quickly recognizes there appears to be red thing and a blue thing, and then
 they appear to be together.
 And then in the next thing, you see, oh, there appears to be a blue thing and the red thing
 in the next thing, they appear to be together.
 And if you look here, it always appears to be the red thing going to the blue thing in
 the most direct way.
 So in the in the along the grid.
 That's all that the human needs to see two examples and the human most humans will already
 make that inference and can now solve if there is like, if there now is a test example, where
 the blue thing is like, the blue thing is down here.
 And the red thing is here like this.
 And it asks you what comes next, you know, you know that the red thing is going down
 to the blue thing.
 But it's very hard to train a machine to do this.
 So I like this test, because it's sort of a different test.
 And I believe the test these tests weren't procedurally generated.
 These tests were actually generated by sholey or, you know, by by actual humans.
 That's pretty cool.
 And 1000 tasks like this is going to be very hard to solve.
 There are even more abstract priors like goal directedness.
 So now you here you can already see this a little bit in that you can say, well, the
 red thing wants to go to the blue thing.
 So there is a notion of time involved, maybe.
 There's also counting and numbers and numbers prior.
 So here you see like a time process.
 So in this demonstration example, you see blue things here, red, big thing.
 And then the next the output grid is this green thing.
 And as a human immediately recognize, okay, so it shoots out from the blue thing, the
 green thing shoots from the blue thing, hits the red wall and goes here.
 Try to make a machine understand this.
 This is insane, right?
 So if you look at the more examples, it all it appears that the blue thing always comes
 from somewhere like the side of the image, and the green thing comes out obviously from
 whatever is not at the at the border of the image and then bounces off the red thing if
 it hits the red thing.
 Now here you can you can already see what's going to happen.
 Remember your AI would need to first determine aha, okay, all of these output grids, they
 seem to be the same as the input grid.
 So it would need to explicitly construct the output grid in the same manner as the input
 grid because it understands this right?
 This is not the same in every task, then it needs to recognize the red thing that stays
 in every one.
 So it needs to put the red thing here right from from here.
 And then it needs to recognize the blue thing stays as well.
 And then most most shockingly needs to recognize, okay, I will draw a line in pixels and lines
 in pixels are hard here.
 And then as soon as it would hit the red thing, it bounces off into the other direction.
 So from just these three examples, the machine has to understand that and correctly output
 the exact solution, not an approximate solution, the exact solution.
 Okay, so yeah, there are these basic geometry and topology priors like lines, rectangular
 shapes, symmetries, rotations, translations, shape upscaling, containing being contained,
 drawing lines, connecting points, and so on.
 Now, let's look at some more examples.
 These are fun, right?
 Check out this one here.
 So you see green, red, and then somehow the green connected to the red.
 So this is an example of that has many of these priors in many of these concepts in
 there is goal directedness, you can already sort of form the hypothesis that the green
 wants to go to the red.
 But also you see that somehow it sort of appears to the blue things seem to be maybe obstacles
 and it appears to change direction when it encounters an obstacle like here.
 So here, you see the example, and you probably confirm so your hypothesis could be it always
 goes until it hits and then it changes direction towards the red thing, right?
 Always towards red thing, because it's not always towards the right because you return
 toward the left.
 So it goes somehow towards the red thing and so it's pretty ambiguous in this situation,
 but you can also make the assumption that if it's ambiguous, it goes towards the middle,
 maybe, maybe.
 So here, again, now we're actually confirming probably so we go towards the red thing, which
 would be towards this direction, then we hit an object, then we go towards the red thing
 until we hit an object, and then we go here.
 Also see that these grids here are not the same size.
 So it's not always the case that the grids of within the same tasks are even the same
 size.
 So now here, again, your AI would need to recognize what size of grid it needs to draw
 and what the result is.
 So it would need to copy this entire grid and also change these pixels right here to
 be green pixels.
 That's hard.
 I mean, that's I find I find this to be pretty hard.
 This is the line extrapolation and turning on obstacle and efficiently reaching a goal
 prior.
 That's crazy.
 And is there more?
 Yes, there is two more, I believe.
 Yeah, those are the last examples.
 So in this one, you can see right here, there appear to be objects, which there's this blue
 objects appear to be the same and there are these red, and then the output grid is one
 of these blue objects.
 Okay, so here we again see different objects, the output grid is one of them.
 So as a human, you can already recognize the output grid is probably always going to be
 one of these objects.
 And now we need to decide on which one.
 So we can formulate the hypothesis that it's probably going to be the one that's the most
 like here, there's three of the blue ones here, there's four of the yellow ones, that's
 more than any other.
 And this here confirms our hypothesis that the it's the object that appears most often.
 Now again, see that there is this notion of object ness.
 You need to upscale somehow.
 No, this is not upscale because the grid is the same size.
 It's simply the image that's upscale.
 But you need to somehow focus be able to focus in on one of these objects, I need to count
 them, you need to compare the counts via each other.
 And now here you can pretty easily see that the output grid is going to contain one of
 those blue things as a human.
 And here, it's it's sort of a symmetry filling task.
 Now as a human, you need one demonstration to get this.
 Maybe you need more, but many tasks involve some sort of symmetry.
 Okay, drawing the symmetrized version around the version of a shape around a marker, that's
 going to be fairly hard for a machine to learn without without the developer knowing that
 this task is coming.
 Okay, they highlight some differentiations to standard psychometric tests.
 But what I find interesting here is that this thing, what a solution to arc may look like
 and what it would imply for AI applications.
 They say we have found art to be fully solvable by humans.
 So they set a human in front of every, every one of these tasks, and it's solvable.
 While many arc tasks are intellectually challenging human test takes us appear to be able to solve
 the majority of tasks on their first try without any practice or verbal explanations.
 In effect, in this task, you get three tries at each at each of the at each of the problems,
 you get three, three tries and the humans can already solve it in one.
 So that just shows you shows you how cool humans are.
 So here is a shawley suggests a solution approach says by start by developing a domain specific
 language capable of expressing all possible situations, all possible solution programs
 for any arc task.
 Since the exact set of arc tax is purposely not formally definable, this may be challenging.
 The space of tasks is defined as anything expressible in terms of arc pairs that would
 only involve core knowledge.
 So core knowledge is this set of human priors that we discussed last time like objectness
 and symmetries and geometric shapes and navigation and so on.
 So he asks you to basically develop a DSL that can capture all the different tasks.
 So so kept basically define a formalism of these tasks.
 But it's hard because you don't know what the tasks are going to be.
 So your best bet is probably to make a formalism that completely over represents what the tasks
 can be.
 It would require hard coding the core knowledge priors from 3.1.2 in a sufficiently abstract
 and combinable program form to serve as a basis functions for a kind of human like reasoning
 DSL.
 We believe that solving this specific subproblem is critical to a to general AI progress.
 Basically says whenever we can describe this is like saying that this AI progress will
 make a big step once we can formally describe human priors.
 And while true this I feel the hardness of this problem is as hard as actually building
 general artificial intelligence or very close to it.
 So it is a bit of a like how to how to go how to build a GI step one build a GI that's
 sort of and not exactly but it's kind of what this says.
 Right.
 So if I could actually have this DSL to describe every single task and I could do it you know
 such that it is not not super over capturing all the tasks then I would be able and I would
 have described human core knowledge in a sufficiently accurate degree that I could just you know
 build a GI.
 So he goes on says given a task use the DSL to generate a set of candidate programs that
 turn the input grids into the corresponding output grids.
 This step would reuse and recombine sub programs that previously proved useful in other arc
 tasks.
 So says whenever you have captured the core knowledge or whenever you have captured the
 problem space in a formal language you can simply use that formal language to express
 whatever your input is so the that turn the input grids into the corresponding output
 grids.
 So you would put in these demonstration examples and describe this with your formal language
 that you have and you can somehow reuse and recombine sub programs that previously proved
 useful for basically asking you to write to come up with source code that would generate
 these demonstration examples in the language of your DSL.
 And then he says select top candidates among these programs so you would generate multiple
 versions of source code that generate these things based on a criterion such as a program
 simplicity or program likelihood.
 Note that we do not expect that merely selecting the simplest possible program that works on
 training pairs will generalize well to test pairs.
 And then use the top three candidates to generate output grids for the test examples.
 So I hope the approach here I feel it makes sense but it is sort of over hopeful in my
 mind and that's mainly because of step one.
 So step one asks you to come up with like a programming language that can capture all
 the tasks in this all the tasks in the data set even though you don't know what the tasks
 are and that has this human core knowledge in inside of it in a in a formally describable
 way.
 And then once you have that programming language you would if you're given this task where
 you have you know a bunch of these demonstration you have a bunch of these demonstration things
 and then you have the test thing you would generate all the programs that would produce
 these demonstration examples or that would given the demo given the input grade would
 produce the output grid right you would generate all the programs and then you would select
 somehow among all these programs the one that you think generalizes the most and you would
 use that program to put this in and get out the solution.
 They say it's probably it's not always the simplest program not always the shortest program
 maybe who knows like I feel step one is the kind of the crucial issue here.
 Okay so they say they make some claims here and about what this what this would bring
 the community we posit that the existence of human level arc solver would represent
 the ability to program and AI from demonstrations alone only requiring a handful of demonstrations
 to specify complex tasks to do a wide range of human relatable tasks of a kind that would
 normally require human level human like fluid intelligence.
 As supporting evidence we note that human performance on psychometric intelligent test
 which are similar to our is predictive of success across all human cognitive tasks.
 Further we posit that since an arc solver and human intelligence would be both founded
 on the same knowledge priors the scope of application of an arc solver would be closer
 to that of human cognition making it such a solver both practically valuable and easy
 to interact with and would produce behavior that is in line with human expectations.
 Okay so they're making the same argument that anyone before has made but they condition
 it on some things and this is I think the conclusion of the entire article here of on
 the measure of intelligence because people had this hope and they say that here claims
 are highly speculative and may prove incorrect much like Newell's 1973 hopes that progress
 on chess playing would translate into meaningful progress and achieving a broad range of cognitive
 abilities especially if arc turns out to feature unforeseen vulnerabilities to unintelligent
 shortcuts.
 This is the AI effect and basically means that whenever you think a task the solving
 of a task represents AI and then you actually see the solution then the solution turns out
 to be not AI in the eyes of the human.
 So the human at first they would say oh this task really requires intelligence and then
 someone solves the task and they'll say oh that's not intelligence you kind of hacked
 your way to that and the expectation is that in this arc challenge there might be a hacky
 way to that but I mean the good question is when at what is there even a task like this
 arc challenge here could that is there even a possibility of a task where you wouldn't
 say that and I'm not so sure about this they seem to be more hopeful than I am but at least
 they say the arc challenge is founded on the same priors as a human has it gives you the
 same amount of experience as a human has right and therefore it is much more comparable to
 human intelligence.
 They go over some weaknesses right here of criticizing their own thing generalization
 is not quantified so they have a measure of generalization in the previous chapter but
 they don't use it right here test validity is not established data set size and diversity
 may be limited and so on but I in my mind this I would not consider this as like an
 AGI task or anything like this I'm pretty sure the solution to this will come in a form
 again where people don't really think it's it exhibits intelligence but I do like the
 task as such and as a machine learner I am very excited to think about how machine learning
 can go about solving this task and especially with what we've seen from something like GPT-3
 it has exactly this kind of structure where you train on a giant data set blah blah blah
 you pre train your language model but then at inference time you input a bunch of these
 demonstration examples and you ask it for the next output so I feel that might be a
 good start for for doing it the question of course is what what then do you pre train
 this model on this GPT-3 for ARC what's the pre training data set for it and I guess that's
 going to be the challenge and probably going to require people to specifically program
 all of these priors into a data set generator for pre training so that would be my approach
 my approach would be write a data set generator for pre training and GPT-3 model to do these
 kind of tasks and in order to write the data set generator you'd have to basically program
 in all of these priors and that's not going to be easy because your best bet is to sort
 of put yourself into the shoes of Chalet and be like oh if I were to design a task what
 kind of things would I do and then try to capture that that's going to be your best
 bet your most honest bet with respect to the challenge is to try to as faithfully as possible
 implement something like an object Ness prior where cohesion and persistence are captured
 that would be the most scientifically sound approach to my approach alright so that was
 my take on the ARC data set if you have any comments I'm very excited to hear comments
 on this if you have already tried the ARC challenge have some insight I also welcome
 comments on that and with that I'll see you next time bye bye