Deep Learning for Symbolic Mathematics
deep learning machine learning nlp natural language processing machine translation arxiv attention mechanism attention transformer rnn recurrent seq2seq facebook fair research math integral ode
This model solves integrals and ODEs by doing seq2seq!

https://arxiv.org/abs/1912.01412
https://ai.facebook.com/blog/using-neural-networks-to-solve-advanced-mathematics-equations/

Abstract:
Neural networks have a reputation for being better at solving statistical or approximate problems than at performing calculations or working with symbolic data. In this paper, we show that they can be surprisingly good at more elaborated tasks in mathematics, such as symbolic integration and solving differential equations. We propose a syntax for representing mathematical problems, and methods for generating large datasets that can be used to train sequence-to-sequence models. We achieve results that outperform commercial Computer Algebra Systems such as Matlab or Mathematica.

Authors: Guillaume Lample, François Charton

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there! Can you solve this? Neither can I. But Wolfram Alpha can. So this is the thing that probably I have most to thank for for passing university, especially the math classes in it. If you don't know Wolfram Alpha, it is an engine from the creators of Mathematica, but it is online. It can do symbolic math. So it can integrate this expression, for example, and you'll get the solution here. And if you have the pro version, it can even give you a step-by-step solution of how to get there. So this part of math is an entirely different part than we usually do with computers. Usually we do numeric math, working with actual values. But here it's about symbolic math. It's about manipulating expression, in this case integrating them. So here is a paper by Facebook AI Research called Deep Learning for Symbolic Mathematics by Guillaume Lampe and François Gertot. These people have basically tackled the task of doing these mathematical reasoning, solving these mathematical problems in a symbolic way using neural networks. So they start out by saying here, neural networks have a reputation for being better at solving statistical or proximate problems. That's what I meant by numeric, then at performing calculations or working with symbolic data. And in this case, they go about this other than other people have. So let's look at how they did it. We can express symbolic mathematics in these kind of trees. So an expression like these up here would be expressed into this tree. So you would have a plus, this 2 plus 3. Sorry, of course there's an implicit bracket here. So you'd have this plus right here, the 2 here and the entire right hand side here. So you can basically decompose it into trees like this or this or this. Here you also can have the differentiation operator as a symbol in there, just like any other operator. Moreover, you can basically decompose everything into everything they have here, into binary and unary nodes in a tree. What that means is either like a plus sign, it has two components, so a left and a right hand side that should be added together. Or like the cosine, it has one argument, namely the thing that it should take the cosine of. So a lot of people have tried going about this problem by working with these trees and basically training neural networks to... So first they use kind of a parser to decompose such a thing into a tree like this. And then use neural networks, let's say tree recursive neural networks or so on, to kind of make sense of the tree and solve it in a recursive manner or something like this. But that has its limitations. So what these people from Facebook AI did is they viewed it as a natural language expression problem. So they say, no, no, let's actually go with trees as sequences. So you can see that this mathematical expression, for example, is already a sequence. It's simply a sequence of tokens. But there are many different ways of expressing this. So you can say 2 plus 3 times the parentheses, you can say 3 times parentheses plus 2. You can turn many things around and there's always these parentheses make it harder and so on. So what they do is they say, OK, let's actually go from this thing to a tree. So let's go to the tree representation and then let's take the tree representation because the tree representation can be normalized. And then let's put that again into a sequence representation such as this one. And this is called reverse polish notation. And it has multiple advantages over the old expression. So let's keep that on the right hand side here. This is the same thing, except it's what is called a prefix notation, whereas the thing on the right here is called an infix notation. Infix because the operators such as the plus is always between its arguments. So it's left hand argument and it's right hand argument. In prefix notation, the operator is always in front of its arguments. So this operator here is has a first argument. This end as a second argument. This right now, the cool thing is if you express a tree like this, you can simply go and use a stack machine to solve it. So you can basically go. I would say you can go from the from the right here and you see you select two and five plus. And let's do it by hand. Actually, this is fun. So we have plus two times three. If you're a boomer like me, you remember you have to use calculators like this and couldn't use the infix notation. So you go from the right, right? You say two, five plus. Cool. That's seven. So scratch that. Put seven here, right? So your new stack is three, two times. This right. Then you go again from the right and you go seven, three times. OK, that's twenty one. Cool. Twenty one. Scratch this. Now it's twenty one. Two plus twenty one is twenty three. I'm fairly sure that's the solution. Well, correct me if I'm wrong. But this is how you would would go about solving like this. So it is the same expression as the original one, but it doesn't use any parentheses. And it is it is derived from the from the tree, basically. So it is you can you can normalize it much more in order to find unique expressions. So what this system does is it it transforms any expression into a prefix notation such as this one. Oops. And then it uses a sequence to sequence model. In order to derive the solution. Now, just how crazy is this? Right. So we come we go from this thing here, right? From this thing. And the solution is twenty one. Right. And the neural network is simply trained to do sequence to sequence from this to that sequence to sequence. That means it basically parses this as a token level. Right. And then it outputs these tokens without. So during training, you simply give it the you give it the input here and you give it the output. And it's supposed to learn how to transform one into the other without you giving it any sort of mathematical ability. Right. Without you telling it what does a plus sign mean without you telling it this algorithm that I just told you. Now, this by itself is already pretty astounding that you would try such a thing. It really transforms the string. So this is not the mathematical equation, but the string of this into the string of that. Now, they don't do it on numbers. Like, I don't think that would work as well if you were to to make it kind of calculate numerical things like this. As we said, this is symbolic. So what it can do is it can, for example, integrate. So if you have an expression like. Let's see some on the bottom here. So if you had an expression such as a polynomial. Here, an expression like this. Right. You would like to find its integral. That is a problem. That's one of the problems we had at the beginning. Right. This integral right here. You can write this in a string like we said. And then derive its solution right here. And have the neural network learn to map one to the other, right, to map this to that. So the way it goes is it would map this into map this into its tree representation. It would map this into its prefix notation. Right. It would also map this to. Let's take another color here. This into its tree. Then it will map this into its prefix notation. And then that's the training data. The training data is take this, derive that. Right. And at inference time, of course, you won't have this here. You'll simply be asked to output a sequence as a normal natural language. Like you can think of machine translation. This thing translates problems into solutions. It's crazy. I mean, it's not it's not technically super challenging, but it's crazy that it works or that it could work. Right. So we'll see how this actually works. They use a transformer model, which is just which is a classic model. If you don't know what a transformer is, I have a video called Attention is All You Need about transformers. You can basically use them to do these kinds of tasks, to map one string into another string. So. Yeah, so they go into detail here of how they construct the data set and how big the problem space is and so on. Ultimately, they compare their system to. Mathematica, I think, and Maple and MathLab, which do the same thing. So Mathematica, which is the kind of desktop version of Wolfram Alpha that I've shown you, you have it here. So integration. Is the task of integrating, let's say, these these symbolic expressions. ODE order one and order two are slightly different tasks where you're asked to solve an ordinary differential equation, which is also a task in symbolic mathematics. If you compare it to Mathematica here and they give it Mathematica a limit of 30 seconds, what Mathematica will do is it will kind of search the manipulations that it knows. So the advantage of this is it can always give you, let's say, a step by step solution if it finds a solution. Right. It will just start and it will do a tree search, manipulating the expression you give in until it reaches a satisfactory solution. But then once it has that, it can give you a path through the tree, which leads to the solution, which will give you a step by step solution. So you can understand it. The system that Facebook designs here doesn't do that. It simply takes right. It simply takes the input tokens like this is the input and it just gives you an output that is learned so that the network per se doesn't understand math. It simply learns from many, many examples that to transform to to come up with good hypotheses. So if you compare here, Mathematica, for example, it can integrate 84 percent of the things that they put into it. It's not said whether it gets it wrong or simply times out in the rest. I would say it times out because probably Mathematica never gets it wrong because it's an actual symbolic manipulator with defined rules. So I guess the rest of the rest 16 percent, it simply times out, doesn't find a solution. Whereas this Facebook system and they say it usually finds the solution in less than a second, finds these solutions in 98.4 percent of the time with a beam size of one. Now, what does the beam size mean? It means that the time that you have to generate the output is the time that you generate the output. So if you have a sequence of input, you can always choose to do a beam search. So when you have a sequence of input, let's actually give an example, a cat jumps. The task is simply to continue the sentence, right, to continue the sentence so you can generate an output sequence. The output sequence could be over the dog. What you can do is you can this is beam size, would be called beam size one or no beam search at all. You can do what's called a beam search in that each step you actually generate multiple hypotheses and then keep the best ones in memory. So you in a beam size of 10, you would always consider the 10 most probable solutions and you would kind of evaluate all 10 and then always keep the 10 best. Let's see how this goes. Let's do a beam size of three in our case. So a cat jumps and then you could come up with three different things. This sentence could continue cat jumps over a cat jumps between and a cat jumps swiftly. Right. So these are your three hypotheses. Then we go to the next step. We have to evaluate each of those, each of them. So a cat jumps over the over a over me. A cat jumps between the between two and a cat jumps between many. The cat jumps swiftly end of sentence, that jumps swiftly over cat jumps swiftly. And right, these are all valid. So of these nine, you would now select again the three that overall have the highest likelihood. Maybe that's the following cat jumps over the cat jumps over a and a cat jumps between two. These three. Right. So you just keep these three. And then in the next step, you again from these three, you would want for each three hypotheses and so on. So this is what's called a beam search. And if you give it a beam size of 10 or 50, this system tends to improve even more. The way this system works is quite different from Mathematica in that Mathematica, as I said, is a symbolic solver that never makes mistakes, but can fail to give you a solution. This system simply generates an output sequence that is not guaranteed to be actually a solution to the problem. It's just a hypothesis. But then you can quickly check whether the hypothesis is correct. So the nature of these math problems with integration, you can simply differentiate. And with ODE, you can simply plug them in to see if there is solution. It's kind of like your classic, let's say, NP-hard problems or like a SAT solving where you can quickly check whether something is a solution. So if you have a system that generates 50 hypotheses, you could quickly check which one is actually correct. So these numbers here mean that one of these 50 that the system came up with was a correct solution. And if you allow for such many hypotheses, you can see it goes up quite a bit. For example, the ODE solving is almost the same. And here it's even worse if you take ODE's of order 2. It's even worse than Mathematica. But if you allow for larger beam sizes, you see it dramatically goes up. And so it's a different approach. I wouldn't be surprised if Mathematica would actually implement something like this very soon or just buy this off of Facebook or something, or Facebook by Mathematica in whatever way. This clearly is a different approach and it appears to work better. But there is a caveat. So here's the caveat that I see with this kind of thing. These evaluations are done on data sets, of course. And this paper goes into big detail on how to generate these data sets. So they have to pay attention to many things like many solutions are equivalent. For example, here, you know, that this solution and this solution to this equation, to this differential equation are the same. So they have to use a symbolic framework to check whether the solutions are the same and so on. This it is very good work, but they do evaluate on expressions that fit into their data set. So here in their data set, they say, OK, we evaluate, you know, expressions with up to 15 internal nodes, 11 leave values for these four binary operators, then these 15 unary operators. So the expressions that they train on fall into this data set. Right. Also, just numbers from negative five to five. So it is it is kind of to be expected that a system that is trained on these things would meet would perform very well on these things as opposed to opposed to Mathematica. That is, you know, a general purpose tool. Moreover, if you look at. Sorry, I think this is further down. For example, in integration for the integration task, they have three different ways of solving of generating data. They have the forward way where they simply use a symbolic integrator to generate expressions. They have the backward way where they start from the integral and then differentiate it in order to obtain a training pair. And they have an integration by parts method. These are three different methods to come up with problems for this system to be trained on. And they have very different properties to the effect that if you train with one just one, it won't work well on the other. So if you train with the forward method, it will work very well on data that has been generated with the forward method. So this is down here. This is what it's trained on. And this is what it's evaluated on. Right. You can see the diagonal is very, very strong. But if you train with the backward method, but you evaluate on data generated with the forward method, it is actually very poor. That's because in one case, generally, the solutions are longer than the input. In the other case, the solutions are shorter. So not only does this system only work on the particular task here, it is actually very attuned to the way that this data was generated. Right. So in fact, I would postulate that this training data is only probably a very small subset of all of the things that we would like to integrate. And again, the problem the problem is made kind of worse because they their evaluation set would also come from their distribution. So what they've ultimately shown is that they can do this on a very skewed, probably very biased subset of this mathematical problem. And on that biased subset, they can outperform something like Mathematica. Right. They kind of defeat themselves. Yeah. If you look here, they even the different integration data generating methods, if you only train on one of them, it doesn't generalize. If you only train on forward data, then if you evaluate on backward generated data, it doesn't work. So even the integrator can't really generalize. So they have to kind of combine different method. And even now, we can probably easily find examples that this integrator can't solve. So, I mean, there is a lot of cool things here and they show a number of properties that the model learns just from without them telling it to. And it's cool that it works anyway. As I said, this model has no programmed in notion of how math works. But also it kind of shows the problems if you do this via a training data set in that if your training data set is very skewed and then your evaluation set follows the same generation process, the claims you can make at the end are limited. And to be fair, I don't know what claims they made in the press generally. So I think there is a pretty cool work. Check it out. And that was it. Thanks.
 Hi there! Can you solve this? Neither can I. But Wolfram Alpha can. So this is the thing that probably I have most to thank for for passing university, especially the math classes in it.
 If you don't know Wolfram Alpha, it is an engine from the creators of Mathematica, but it is online. It can do symbolic math. So it can integrate this expression, for example, and you'll get the solution here.
 And if you have the pro version, it can even give you a step-by-step solution of how to get there. So this part of math is an entirely different part than we usually do with computers.
 Usually we do numeric math, working with actual values. But here it's about symbolic math. It's about manipulating expression, in this case integrating them.
 So here is a paper by Facebook AI Research called Deep Learning for Symbolic Mathematics by Guillaume Lampe and François Gertot.
 These people have basically tackled the task of doing these mathematical reasoning, solving these mathematical problems in a symbolic way using neural networks.
 So they start out by saying here, neural networks have a reputation for being better at solving statistical or proximate problems.
 That's what I meant by numeric, then at performing calculations or working with symbolic data.
 And in this case, they go about this other than other people have. So let's look at how they did it.
 We can express symbolic mathematics in these kind of trees. So an expression like these up here would be expressed into this tree.
 So you would have a plus, this 2 plus 3. Sorry, of course there's an implicit bracket here.
 So you'd have this plus right here, the 2 here and the entire right hand side here.
 So you can basically decompose it into trees like this or this or this.
 Here you also can have the differentiation operator as a symbol in there, just like any other operator.
 Moreover, you can basically decompose everything into everything they have here, into binary and unary nodes in a tree.
 What that means is either like a plus sign, it has two components, so a left and a right hand side that should be added together.
 Or like the cosine, it has one argument, namely the thing that it should take the cosine of.
 So a lot of people have tried going about this problem by working with these trees and basically training neural networks to...
 So first they use kind of a parser to decompose such a thing into a tree like this.
 And then use neural networks, let's say tree recursive neural networks or so on, to kind of make sense of the tree and solve it in a recursive manner or something like this.
 But that has its limitations.
 So what these people from Facebook AI did is they viewed it as a natural language expression problem.
 So they say, no, no, let's actually go with trees as sequences.
 So you can see that this mathematical expression, for example, is already a sequence.
 It's simply a sequence of tokens.
 But there are many different ways of expressing this.
 So you can say 2 plus 3 times the parentheses, you can say 3 times parentheses plus 2.
 You can turn many things around and there's always these parentheses make it harder and so on.
 So what they do is they say, OK, let's actually go from this thing to a tree.
 So let's go to the tree representation and then let's take the tree representation because the tree representation can be normalized.
 And then let's put that again into a sequence representation such as this one.
 And this is called reverse polish notation.
 And it has multiple advantages over the old expression.
 So let's keep that on the right hand side here.
 This is the same thing, except it's what is called a prefix notation, whereas the thing on the right here is called an infix notation.
 Infix because the operators such as the plus is always between its arguments.
 So it's left hand argument and it's right hand argument.
 In prefix notation, the operator is always in front of its arguments.
 So this operator here is has a first argument.
 This end as a second argument.
 This right now, the cool thing is if you express a tree like this,
 you can simply go and use a stack machine to solve it.
 So you can basically go.
 I would say you can go from the from the right here and you see you select two and five plus.
 And let's do it by hand.
 Actually, this is fun. So we have plus two times three.
 If you're a boomer like me, you remember you have to use calculators like this and couldn't use the infix notation.
 So you go from the right, right?
 You say two, five plus. Cool.
 That's seven. So scratch that.
 Put seven here, right?
 So your new stack is three, two times.
 This right.
 Then you go again from the right and you go seven, three times.
 OK, that's twenty one. Cool.
 Twenty one. Scratch this.
 Now it's twenty one. Two plus twenty one is twenty three.
 I'm fairly sure that's the solution.
 Well, correct me if I'm wrong.
 But this is how you would would go about solving like this.
 So it is the same expression as the original one, but it doesn't use any parentheses.
 And it is it is derived from the from the tree, basically.
 So it is you can you can normalize it much more in order to find unique expressions.
 So what this system does is it it transforms any expression into a prefix notation such as this one.
 Oops. And then it uses a sequence to sequence model.
 In order to derive the solution.
 Now, just how crazy is this? Right.
 So we come we go from this thing here, right?
 From this thing. And the solution is twenty one.
 Right. And the neural network is simply trained to do sequence to sequence from this to that sequence to sequence.
 That means it basically parses this as a token level. Right.
 And then it outputs these tokens without.
 So during training, you simply give it the you give it the input here and you give it the output.
 And it's supposed to learn how to transform one into the other without you giving it any sort of
 mathematical ability. Right.
 Without you telling it what does a plus sign mean without you telling it this algorithm that I just told you.
 Now, this by itself is already pretty astounding that you would try such a thing.
 It really transforms the string.
 So this is not the mathematical equation, but the string of this into the string of that.
 Now, they don't do it on numbers.
 Like, I don't think that would work as well if you were to to make it kind of calculate numerical things like this.
 As we said, this is symbolic.
 So what it can do is it can, for example, integrate.
 So if you have an expression like.
 Let's see some on the bottom here.
 So if you had an expression such as a polynomial.
 Here, an expression like this.
 Right. You would like to find its integral.
 That is a problem. That's one of the problems we had at the beginning.
 Right. This integral right here.
 You can write this in a string like we said.
 And then derive its solution right here.
 And have the neural network learn to map one to the other, right, to map this to that.
 So the way it goes is it would map this into map this into its tree representation.
 It would map this into its prefix notation.
 Right. It would also map this to.
 Let's take another color here.
 This into its tree.
 Then it will map this into its prefix notation.
 And then that's the training data.
 The training data is take this, derive that.
 Right. And at inference time, of course, you won't have this here.
 You'll simply be asked to output a sequence as a normal natural language.
 Like you can think of machine translation.
 This thing translates problems into solutions.
 It's crazy.
 I mean, it's not it's not technically super challenging, but it's crazy that it works or that it could work.
 Right.
 So we'll see how this actually works.
 They use a transformer model, which is just which is a classic model.
 If you don't know what a transformer is, I have a video called Attention is All You Need about transformers.
 You can basically use them to do these kinds of tasks, to map one string into another string.
 So.
 Yeah, so they go into detail here of how they construct the data set and how big the problem space is and so on.
 Ultimately, they compare their system to.
 Mathematica, I think, and Maple and MathLab, which do the same thing.
 So Mathematica, which is the kind of desktop version of Wolfram Alpha that I've shown you, you have it here.
 So integration.
 Is the task of integrating, let's say, these these symbolic expressions.
 ODE order one and order two are slightly different tasks where you're asked to solve an ordinary differential equation, which is also a task in symbolic mathematics.
 If you compare it to Mathematica here and they give it Mathematica a limit of 30 seconds, what Mathematica will do is it will kind of search the manipulations that it knows.
 So the advantage of this is it can always give you, let's say, a step by step solution if it finds a solution.
 Right. It will just start and it will do a tree search, manipulating the expression you give in until it reaches a satisfactory solution.
 But then once it has that, it can give you a path through the tree, which leads to the solution, which will give you a step by step solution.
 So you can understand it.
 The system that Facebook designs here doesn't do that.
 It simply takes right.
 It simply takes the input tokens like this is the input and it just gives you an output that is learned so that the network per se doesn't understand math.
 It simply learns from many, many examples that to transform to to come up with good hypotheses.
 So if you compare here, Mathematica, for example, it can integrate 84 percent of the things that they put into it.
 It's not said whether it gets it wrong or simply times out in the rest.
 I would say it times out because probably Mathematica never gets it wrong because it's an actual symbolic manipulator with defined rules.
 So I guess the rest of the rest 16 percent, it simply times out, doesn't find a solution.
 Whereas this Facebook system and they say it usually finds the solution in less than a second, finds these solutions in 98.4 percent of the time with a beam size of one.
 Now, what does the beam size mean?
 It means that the time that you have to generate the output is the time that you generate the output.
 So if you have a sequence of input, you can always choose to do a beam search.
 So when you have a sequence of input, let's actually give an example, a cat jumps.
 The task is simply to continue the sentence, right, to continue the sentence so you can generate an output sequence.
 The output sequence could be over the dog.
 What you can do is you can this is beam size, would be called beam size one or no beam search at all.
 You can do what's called a beam search in that each step you actually generate multiple hypotheses and then keep the best ones in memory.
 So you in a beam size of 10, you would always consider the 10 most probable solutions and you would kind of evaluate all 10 and then always keep the 10 best.
 Let's see how this goes. Let's do a beam size of three in our case.
 So a cat jumps and then you could come up with three different things.
 This sentence could continue cat jumps over a cat jumps between and a cat jumps swiftly.
 Right. So these are your three hypotheses.
 Then we go to the next step. We have to evaluate each of those, each of them.
 So a cat jumps over the over a over me.
 A cat jumps between the between two and a cat jumps between many.
 The cat jumps swiftly end of sentence, that jumps swiftly over cat jumps swiftly.
 And right, these are all valid.
 So of these nine, you would now select again the three that overall have the highest likelihood.
 Maybe that's the following cat jumps over the cat jumps over a and a cat jumps between two.
 These three. Right. So you just keep these three.
 And then in the next step, you again from these three, you would want for each three hypotheses and so on.
 So this is what's called a beam search.
 And if you give it a beam size of 10 or 50, this system tends to improve even more.
 The way this system works is quite different from Mathematica in that Mathematica, as I said, is a symbolic solver that never makes mistakes,
 but can fail to give you a solution.
 This system simply generates an output sequence that is not guaranteed to be actually a solution to the problem.
 It's just a hypothesis.
 But then you can quickly check whether the hypothesis is correct.
 So the nature of these math problems with integration, you can simply differentiate.
 And with ODE, you can simply plug them in to see if there is solution.
 It's kind of like your classic, let's say, NP-hard problems or like a SAT solving where you can quickly check whether something is a solution.
 So if you have a system that generates 50 hypotheses, you could quickly check which one is actually correct.
 So these numbers here mean that one of these 50 that the system came up with was a correct solution.
 And if you allow for such many hypotheses, you can see it goes up quite a bit.
 For example, the ODE solving is almost the same.
 And here it's even worse if you take ODE's of order 2.
 It's even worse than Mathematica.
 But if you allow for larger beam sizes, you see it dramatically goes up.
 And so it's a different approach.
 I wouldn't be surprised if Mathematica would actually implement something like this very soon or just buy this off of Facebook or something, or Facebook by Mathematica in whatever way.
 This clearly is a different approach and it appears to work better.
 But there is a caveat.
 So here's the caveat that I see with this kind of thing.
 These evaluations are done on data sets, of course.
 And this paper goes into big detail on how to generate these data sets.
 So they have to pay attention to many things like many solutions are equivalent.
 For example, here, you know, that this solution and this solution to this equation, to this differential equation are the same.
 So they have to use a symbolic framework to check whether the solutions are the same and so on.
 This it is very good work, but they do evaluate on expressions that fit into their data set.
 So here in their data set, they say, OK, we evaluate, you know, expressions with up to 15 internal nodes, 11 leave values for these four binary operators, then these 15 unary operators.
 So the expressions that they train on fall into this data set.
 Right. Also, just numbers from negative five to five.
 So it is it is kind of to be expected that a system that is trained on these things would meet would perform very well on these things as opposed to opposed to Mathematica.
 That is, you know, a general purpose tool.
 Moreover, if you look at.
 Sorry, I think this is further down.
 For example, in integration for the integration task, they have three different ways of solving of generating data.
 They have the forward way where they simply use a symbolic integrator to generate expressions.
 They have the backward way where they start from the integral and then differentiate it in order to obtain a training pair.
 And they have an integration by parts method.
 These are three different methods to come up with problems for this system to be trained on.
 And they have very different properties to the effect that if you train with one just one, it won't work well on the other.
 So if you train with the forward method, it will work very well on data that has been generated with the forward method.
 So this is down here. This is what it's trained on.
 And this is what it's evaluated on.
 Right. You can see the diagonal is very, very strong.
 But if you train with the backward method, but you evaluate on data generated with the forward method, it is actually very poor.
 That's because in one case, generally, the solutions are longer than the input.
 In the other case, the solutions are shorter.
 So not only does this system only work on the particular task here,
 it is actually very attuned to the way that this data was generated.
 Right. So in fact, I would postulate that this training data is only probably a very small subset of all of the things that we would like to integrate.
 And again, the problem the problem is made kind of worse because they their evaluation set would also come from their distribution.
 So what they've ultimately shown is that they can do this on a very skewed, probably very biased subset of this mathematical problem.
 And on that biased subset, they can outperform something like Mathematica.
 Right. They kind of defeat themselves.
 Yeah. If you look here, they even the different integration data generating methods, if you only train on one of them, it doesn't generalize.
 If you only train on forward data, then if you evaluate on backward generated data, it doesn't work.
 So even the integrator can't really generalize.
 So they have to kind of combine different method. And even now, we can probably easily find examples that this integrator can't solve.
 So, I mean, there is a lot of cool things here and they show a number of properties that the model learns just from without them telling it to.
 And it's cool that it works anyway. As I said, this model has no programmed in notion of how math works.
 But also it kind of shows the problems if you do this via a training data set in that if your training data set is very skewed and then your evaluation set follows the same generation process, the claims you can make at the end are limited.
 And to be fair, I don't know what claims they made in the press generally.
 So I think there is a pretty cool work. Check it out. And that was it. Thanks.