TAPAS: Weakly Supervised Table Parsing via Pre-training (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper bert nlp natural language processing wikitables sql tabular aggregations structured google
Answering complex questions about tabular information is hard. No two tables are alike and sometimes the answer you're looking for is not even in the table and needs to be computed from a subset of the cells. Surprisingly, this model can figure it all out by itself through some clever input encoding and loss engineering.

Paper: https://arxiv.org/abs/2004.02349
Code: https://github.com/google-research/tapas

Abstract:
Answering natural language questions over tables is usually seen as a semantic parsing task. To alleviate the collection cost of full logical forms, one popular approach focuses on weak supervision consisting of denotations instead of logical forms. However, training semantic parsers from weak supervision poses difficulties, and in addition, the generated logical forms are only used as an intermediate step prior to retrieving the denotation. In this paper, we present TAPAS, an approach to question answering over tables without generating logical forms. TAPAS trains from weak supervision, and predicts the denotation by selecting table cells and optionally applying a corresponding aggregation operator to such selection. TAPAS extends BERT's architecture to encode tables as input, initializes from an effective joint pre-training of text segments and tables crawled from Wikipedia, and is trained end-to-end. We experiment with three different semantic parsing datasets, and find that TAPAS outperforms or rivals semantic parsing models by improving state-of-the-art accuracy on SQA from 55.1 to 67.2 and performing on par with the state-of-the-art on WIKISQL and WIKITQ, but with a simpler model architecture. We additionally find that transfer learning, which is trivial in our setting, from WIKISQL to WIKITQ, yields 48.7 accuracy, 4.2 points above the state-of-the-art.

Authors: Jonathan Herzig, Paweł Krzysztof Nowak, Thomas Müller, Francesco Piccinno, Julian Martin Eisenschlos

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there, have a look at this table on the left. So in this table, in each row you can see the following things. The name of a wrestler, the number of times that wrestler has been a champion and the combined number of days where that wrestler has been a champion or like the sum of the length of all their championships. Along with the column that is the rank and this is ranked by the combined days attribute. So this table is very interesting by itself but if you look at the right we have a couple of questions and let's try to answer them. Which wrestler had the most number of reigns? So for that you need to go to number of reigns column and you need to mentally sort them and you'll find out that 8 is the highest number and therefore Ric Flair is the wrestler you're looking for. Second question, the average time as a champion for top two wrestlers. Now we need to go the top two wrestlers, we can guess that pertains to the rank and then so we want the average of these two numbers. You even have questions such as which of the following wrestlers were ranked in the bottom three? Which the answers would be all of those and then after that out of these who had more than one reign? And you can see that's Dan Severn. So the paper that we're having here is trying to answer questions like this if given a table. As you can see this is a pretty pretty hard task and so pretty excited to read this. The paper is called TAPAS weekly supervised table parsing via pre-training by Jonathan Herzig, Pavel Kristof Novak, Thomas Miller, Francesco Piccino and Julian Martin Eisenschloss. Full disclaimer I know these people so I might be slightly biased. Alright so you've already seen the task. The task is you are given a table and a question and you're trying to answer that. Now there it's not as easy as that but the table questions come in different forms as you have seen. Sometimes you just need to select a cell from a table like we have here. The first question I simply, most number of reigns, I simply select whatever that is. So the answer here is already in the table, Rick Flair and this they call a cell selection task. This is wherever you need to select a cell. The same for these bottom here so which of the following wrestlers are ranked in the bottom three and out of these which one have more than one reign? All of these answers are in the table somewhere. The second thing is what they call a scalar answer. That is when the answer to be computed is a number that is not in the table. So these average time here which turns out to be 3426 is nowhere to be found in the table. So there actually needs to be a computation performed by the model. And lastly you have these things called ambiguous answers. Now the ambiguous answers refer to a thing where it is a number that you're looking for so how many but the number here is in the table. So you can think of this in terms of training data. If you have a task like this and you have training data and you just have the question, you just have this question and you're given the answer 2. You can teach your model either to select this number here or you can teach your model that would be wrong. Because how many world champions are there with only one reign to simply select this cell here is not correct. Because that cell even though the number is 2 it doesn't mean the same thing. It's not counting right. So the correct program here would be to count the number to count the cells where there is a 1 here which is also 2. And they call this situation ambiguous answer. So you might have already guessed that a single model that does all of this needs to sort of have multiple modes. That's exactly what they propose. So they propose a model that takes in the table and the question. And then in the first step it selects its mode. So the mode is either the cell selection or it is to compute something. And then whenever it's a cell selection it simply has a component to select cells. But when it's a compute it needs to decide in the second step what to compute and then also select the appropriate cells. So this is the model. Now this stuff like this has existed for a long time in these table answering things but the way we want to do it here is end to end with a single deep learning model of course. Because we want to be better than anything else. And the trend in deep learning is to put more and more into one model and to have it end to end differentiable. So you see we need multiple components. We need some sort of a mode selector. We need some sort of a cell collector and we need a thing that decides if we are in the compute mode what computation to be done. Now let me present the model that this paper proposes. So this paper proposes to embed the question here. So you can see here that's the question into a BERT input. So this is a transformer right here. This is BERT or any variant of BERT that you can think of. So the question is embedded as natural language and then interestingly enough the table right here is also embedded as language. We'll get to that in a second. But the question and the table are in the input and then the model is asked to do two things. First of all it's asked to do an aggregation prediction. So this can either be one of these programs called count sum average or it can be as you can see here none. So no aggregation. So this handles our first two components. It can decide to perform a calculation or none and if it is performing a calculation it can decide to do a count sum or an average. Now of course the model here is not limited to those computations. You can think of extending this to any further computation. The important thing is that they have a number as an output. Second of all there is a cell selector. So depending on this aggregation prediction you need some cells. Like if you want to compute an average you need the cells to compute an average over. So the cell selector here will select cells from the table. Specifically it goes by row and column. Sorry column and row. Since these tables usually they have a header right this is the table header where the attributes are listed. It makes sense to first in a first step select which column you want to select from and then if once you have a column let's say this column here in the second step you say which of the cells you want to select. Now these can be multiple but the way the system is set up it's first a column selector and then a cell selector within that column. So you can only ever get columns from the same cell in this thing. Let's remember that for later. Alright so this is what the model does now let's look at the input. The input to the model is this here. Now this if you refer from this before this was in this blue box and then here you'd have the computation selection and here you have the cell selection. So this is this is how you can relate that. So usually if you input something into a transformer what you want to do is you want to embed this into into a token embeddings. So first you want to split everything you put in into what are called tokens. Now tokens are either things like words or word pieces the important thing is to have a dictionary for it and each one gets mapped to a vector. So this here is your query. You take your query as a string and you tokenize it and you get the embeddings from the embedding table and that's your input right. So it's a sequence of token embeddings and then you also embed the table and this I find pretty cool here in this model and somewhat special is that the table is actually presented as just natural language. So you can see here the table is one string it's just a single string that goes from left to right it's just the serialized table. So this table right here you can see these are word pieces so this table if I reconstruct it if I can attempt to reconstruct it it is going to be a table that has as the headers call one call two these are the names so in days before here would be name of the wrestler and this would be number of days. And then here 0 1 2 3 so this this table right here corresponds to this string right here. I hope you can you can make sense of that. So the table is just put there as one long string and then in order to make the model realize you know what the table is you have these special embeddings. So usually in BERT you have what they're called position embeddings to indicate where in the sequence that is. So in a simpler in the simplest case these are embeddings for the numbers 0 1 2 3 4 and so on so wherever the position is. This you can all look up in the attention is all you need video I've made that if you are unfamiliar with transformer inputs. Then also the the segment embeddings simply indicate where what a token is part of. So for every token that's part of the query you see you have segment 0 embedding and for every token that's part of the table you have a segment 1 embedding. This is simply to tell the model hey this particular token is part of the question or part of the table. Then you have the new things so this paper newly introduces the following embeddings column and row embeddings. Now these for the question of course they don't make any sense but you have to put something here so you just put column 0 but for the table you see there is a column 1 and column 2 and the this exactly so we've seen that this here is the header of column 1 and this is the header of column 2 and then it goes back column 1 column 2 column 1 column 2 and you can see here this 0 is in column 1 and this one is in column 2 and this in column 1 again and the same for the rows so you have row 0 for the headers and then row 1 for the first two numbers and row 2 for the second two numbers. So this is all of this so you see these two are in the first row and these two are in the second row. All of this is to tell the model all of this information down here is to tell the model how this table looks so if it wants to select the second column from the third row it would look in this information to see which cell to select and then the last thing they introduce is this so-called rank embeddings. Now as we've seen before if this first column here is maybe the sorry the number of days of something so this is the number of days and this second one is the number of reigns so how many championships the table can only be sorted at maximum by one of them so you want to sort of for each cell you want to tell the model let's extend that table by two numbers 4 and 1 so for each column you want to tell the model the ranking of the numbers so here it's pretty easy this is rank 1 this is rank 2 this is rank 3 further on the left side this is rank 1 this is rank 2 down here and this is rank 3 so the model has an will have an if you give this information the model have an easier time to detect like give me the top 2 or something like this give me the worst give me the best give me the highest and so on the model will have an easier time doing that so that's why the rank here as you can see the zero and as also the number one are embedded rank one and the other two rank two because they're just lower now I don't feel I feel they could could have given a better example than this table I feel you could actually put real names here to make clearer not call one and call two and I feel you could give a somewhat smarter content because if you just look at the picture here you cannot see the correspondence of these rank tokens because in essence they are exactly equal as the row tokens but fortunately we can read the text oh there's the table ha so I have actually I've not seen that but I have discerned it correctly for this particular for this particular input alright I think that's the the half of the magic is how you encode the input in such a thing and this seems to be first of all a pretty cool idea but second of all it exactly is what this kind of new regime of NLP is about is that you basically put everything as a string you annotate it in a smart way and that lets the model figure out a lot of stuff about the input people used to people used to do the very different things so people if given a query and a table like this what people would do is they would somehow first of all get the table headers and and kind of guess the data types of the attributes and then they would formulate reformulate the query maybe also with a neural network maybe with something else into something like SQL in order to actually have an SQL statement to select the correct cells or perform the correct aggregations and that is somewhat brittle and it's just much less deep learning than this model so I like this part of the model now the problem of course is as we've seen in this multi step process so how do we first of all if you build if we want to build a cell selector that's pretty easy right we've seen this so we the cell selector is first column column selection and then second row selection and this can be multiple rows so that's fairly easy selecting cells either for just returning or for aggregation pretty easy but how do we do the actually the aggregation selection is also pretty easy because we can just do a multi class classifier right so the classifier will simply tell us a give us a distribution and then we see okay the sum aggregation is probably here the the what the model wants the real question is how do we train this and how this is trained is what I find really interesting so as we've seen they have training data the training data comes in the form of tables questions and answers as we've seen before we don't know how to get to those answers so when the question is which wrestler had the most number of rains we just know the answer is a Ric flair now they they do again a two-step process for their training data that mimics the two step process of the model so the first step is is the answer a number is the answer a number if no then it is definitely a cell selection task so they if it's not a number they just restrict themselves to selecting cells if the answer is not in the table then that just means that the correct thing is to select no cells and just say I can't answer this question if it is a number then again you have two options so is it in the table if yes we are in a weird situation if no not in table then it is an aggregation so if it is a number that is not in the table that means that the answer is a number there's not in the table that means the answer must be computed via one of these aggregations and if the answer is a number but is in the table then we are in this ambiguous answer setting where the it could be that we need to select the cell but it could also be that the same number by accident is in the table but actually needs to be computed from other numbers and they do this in the most deep blurny way possible is that they do basically a soft decision here so they let the model when they let it select what to compute they let it make a soft decision what do I mean by that so let's say you have these three operations count sum and average and you have the cell selection so the cell selector will basically tell you I will select three cells the three cells contain the number seven the number eight and the number three alright so and the question was I don't even know what the question was but the cell selector tells you these three cells are to be selected you do this by simply selecting the cells where the cell selector has a higher probability than one half now your your aggregation selection module gives you a softmax distribution over over the actions so it's not very much count here maybe that's 0.1 this here is maybe 0.3 and this is the 0.6 what you do is you simply compute all of them so you want to compute the count here which is three you want to compute the sum here which is 18 and then you want to compute the average which is six ha I made a good example by accident and then you simply weigh the outputs here by their probabilities so you say since the model wants point one puts what point one probability on the count I'm going to have 0.1 times 3 plus it wants point three times this so 0.3 times 18 plus 0.6 times 6 now I'm not gonna so this is 6 plus point three plus 3.6 9.9 so that that's how the model computes things it simply puts probability on these operations here and then you simply take a weighted output with respect to the computation of all those things now I'm pretty sure that's completely invalid because for the same numbers for example the sum is going to have a much larger like variance than the average and and that's somewhat going the count maybe somewhere in between depending on the numbers so this just to take the weighted average here and then of course right so what they do is they do have this this is the model output and you have the correct answer let's say the correct answer was actually was to compute the the average so the correct answers six so what they do is simply they take the squared error and that's their loss actually they don't take the squared error they take a approximation to the squared error which is square until some Delta and then it's linear and this is simply to be a bit more outlier robust and they do other things to be more outlier robust but this so this is the model output and this is the correct answer and they simply count on the fact that this will this will back propagate so if you want to make these two things closer if you're the model right you have the option of simply putting more weight from the from the other ones on to the average operation and that will decrease the 9.9 because you as you can see both of these numbers will get smaller and no wait this isn't the yes sorry so you will you will decrease these numbers so this is the output we got from the weighted average right so if we decrease these weights you will put weight from here to here that will bring the number 9.9 down and that will get you closer to the answer you're looking for but you can also achieve this by you can achieve this even more right so this 9.9 is too high if we want to bring the 9.9 down we're much better off by taking some of that output and actually putting on this here because three is the lowest number right the only agreement here is that we want to take weight away from the 18 from the large one so I'm extremely surprised that this works given that it is so super ambiguous what the model should do with these operations and I I highly doubt that you can extend this so it's of course agnostic of what these aggregations are but to be able to extend this to many more aggregations is will I think lead to much more of these situations where the model is entirely unsure of where to put the mass of where to put the weight and I would be interested to see what happens if you have a data set with like 20 or 50 of these aggregations and not just three so this is the this is the let's say the the interesting part here the other if you go the other way when you have this cell selection task it is just to select a cell right and then you simply have the cell selector that part here that does the selection that you also you train every time simply to give each cell a weight right so this this is simply the softmax over column and then the softmax over rows and you can train that using the cross entropy now training this cell selector from data is pretty easy when it's a cell selection task right because the answer is in the table and or is not in the table and then you know to select no cell so you do have the training data that a particular cell is the correct cell and you can train the model to select that cell but it is actually a pretty hard task if it is for example you're looking for an average operation because not only do you are you not really sure that it's an average operation you just know that that kind of gives you the correct answer you also don't really know which cells to select for this average operation right because depending on which cells you select and of course that's going to be a soft selection as well the the average answer the average will be different depending on which cells you select so they're basically counting on this loss here to back propagate not only through the the selection of the aggregation to perform but also to the cell selector to set which cells to to select so from this weak signal it's almost like the reinforcement learning problem where you have the weak signal and you have like a billion ways to get your number closer to that signal and not not really accurate understanding of what you need to do is you're just relying on the model through lots and lots and lots and lots of data to kind of figure out which natural language questions to map to which cell selection and aggregation so this is it's a it seems like impossible but it works the last thing we need to talk about is this ambiguous answer setting and as you can imagine it's pretty simple that they also let the model do and a soft selection between the cell selection tasks so no aggregation and the aggregations to be performed and basically let the model figure out itself which one is better to do an aggregation or to do no aggregation suffice to say this this only works for pretty I think I think it only works for pretty limited amount of tasks pretty limited amount of questions and you might have spotted there even these questions that are follow-up questions which are another thing they build into the model and I don't I'm not really gonna talk about this but they do have this concept as well which I find maybe a bit out of place but maybe it's just part of their data set somewhere maybe it's just these companies want to get into this conversational mode so everything needs to be context dependent at the interesting part here is really the computation of the aggregates and specifically the question of which of these aggregations to choose and this again this is so surprising that it works and fairly fairly cool I think that is the gist of the paper they do extremely thorough evaluations here on these data sets and ablations to see what really counts and what doesn't I don't really want to go into that safe to say their results are better than anything else before I believe they I believe they're actually on par with another model but in one data set but they beat them on every other data set so that's you know that's cool I don't think there was a bar nevermind I invite you to check out this paper look for yourself they have the code online if you want to train a model like this yourself other than that thanks for listening if you like this content please subscribe like comment tell a friend and bye bye
 Hi there, have a look at this table on the left. So in this table, in each row
 you can see the following things. The name of a wrestler, the number of times
 that wrestler has been a champion and the combined number of days where that
 wrestler has been a champion or like the sum of the length of all their
 championships. Along with the column that is the rank and this is ranked by the
 combined days attribute. So this table is very interesting by itself but if you
 look at the right we have a couple of questions and let's try to answer them.
 Which wrestler had the most number of reigns? So for that you need to go to
 number of reigns column and you need to mentally sort them and you'll find out
 that 8 is the highest number and therefore Ric Flair is the wrestler
 you're looking for. Second question, the average time as a champion for top two
 wrestlers. Now we need to go the top two wrestlers, we can guess that pertains to
 the rank and then so we want the average of these two numbers. You even have
 questions such as which of the following wrestlers were ranked in
 the bottom three? Which the answers would be all of those and then after that out
 of these who had more than one reign? And you can see that's Dan Severn. So the
 paper that we're having here is trying to answer questions like this if given a
 table. As you can see this is a pretty pretty hard task and so pretty excited
 to read this. The paper is called TAPAS weekly supervised table parsing via
 pre-training by Jonathan Herzig, Pavel Kristof Novak, Thomas Miller, Francesco
 Piccino and Julian Martin Eisenschloss. Full disclaimer I know these people so
 I might be slightly biased. Alright so you've already seen the task. The task is
 you are given a table and a question and you're trying to answer that. Now there
 it's not as easy as that but the table questions come in different
 forms as you have seen. Sometimes you just need to select a cell from a table
 like we have here. The first question I simply, most number of reigns, I simply
 select whatever that is. So the answer here is already in the table, Rick Flair
 and this they call a cell selection task. This is wherever you need to select a
 cell. The same for these bottom here so which of the following wrestlers are
 ranked in the bottom three and out of these which one have more than one reign?
 All of these answers are in the table somewhere. The second thing is what they
 call a scalar answer. That is when the answer to be computed is a number that
 is not in the table. So these average time here which turns out to be 3426 is
 nowhere to be found in the table. So there actually needs to be a computation
 performed by the model. And lastly you have these things called ambiguous
 answers. Now the ambiguous answers refer to a thing where it is a number that
 you're looking for so how many but the number here is in the table. So you can
 think of this in terms of training data. If you have a task like this and you
 have training data and you just have the question, you just have this question and
 you're given the answer 2. You can teach your model either to select
 this number here or you can teach your model that would be wrong. Because
 how many world champions are there with only one reign to simply select this
 cell here is not correct. Because that cell even though the number is 2 it
 doesn't mean the same thing. It's not counting right. So the correct program
 here would be to count the number to count the cells where there is a 1 here
 which is also 2. And they call this situation ambiguous answer. So you might
 have already guessed that a single model that does all of this needs to sort of
 have multiple modes. That's exactly what they propose. So they propose a model
 that takes in the table and the question. And then in the first step it selects its
 mode. So the mode is either the cell selection or it is to compute something.
 And then whenever it's a cell selection it simply has a component to select
 cells. But when it's a compute it needs to decide in the second step what to
 compute and then also select the appropriate cells. So this is the
 model. Now this stuff like this has existed for a long time in these table
 answering things but the way we want to do it here is end to end with a single
 deep learning model of course. Because we want to be better than anything else. And
 the trend in deep learning is to put more and more into one model and to have
 it end to end differentiable. So you see we need multiple components. We
 need some sort of a mode selector. We need some sort of a cell collector and
 we need a thing that decides if we are in the compute mode what computation to
 be done. Now let me present the model that this paper proposes. So this paper
 proposes to embed the question here. So you can see here that's the question into
 a BERT input. So this is a transformer right here. This is BERT or any variant
 of BERT that you can think of. So the question is embedded as natural language
 and then interestingly enough the table right here is also embedded as language.
 We'll get to that in a second. But the question and the table are in the input
 and then the model is asked to do two things. First of all it's asked to do an
 aggregation prediction. So this can either be one of these programs called
 count sum average or it can be as you can see here none. So no aggregation. So
 this handles our first two components. It can decide to perform a calculation or
 none and if it is performing a calculation it can decide to do a count
 sum or an average. Now of course the model here is not limited to
 those computations. You can think of extending this to any further
 computation. The important thing is that they have a number as an output. Second
 of all there is a cell selector. So depending on this aggregation prediction
 you need some cells. Like if you want to compute an average you need the cells to
 compute an average over. So the cell selector here will select cells from the
 table. Specifically it goes by row and column. Sorry column and row. Since
 these tables usually they have a header right this is the table header where the
 attributes are listed. It makes sense to first in a first step select which column
 you want to select from and then if once you have a column let's say this column
 here in the second step you say which of the cells you want to select. Now these
 can be multiple but the way the system is set up it's first a column selector
 and then a cell selector within that column. So you can only ever get columns
 from the same cell in this thing. Let's remember that for later. Alright so this
 is what the model does now let's look at the input. The input to the model is this
 here. Now this if you refer from this before this was in this blue box and then
 here you'd have the computation selection and here you have the cell
 selection. So this is this is how you can relate that. So usually if you input
 something into a transformer what you want to do is you want to embed this into
 into a token embeddings. So first you want to split everything you put in into
 what are called tokens. Now tokens are either things like words or word pieces
 the important thing is to have a dictionary for it and each one gets mapped
 to a vector. So this here is your query. You take your query as a string
 and you tokenize it and you get the embeddings from the embedding table and
 that's your input right. So it's a sequence of token embeddings and then
 you also embed the table and this I find pretty cool here in this model and
 somewhat special is that the table is actually presented as just natural
 language. So you can see here the table is one string it's just a single string
 that goes from left to right it's just the serialized table. So this table right
 here you can see these are word pieces so this table if I reconstruct it if I
 can attempt to reconstruct it it is going to be a table that has as the
 headers call one call two these are the names so in days before here would be
 name of the wrestler and this would be number of days. And then here 0 1 2 3 so
 this this table right here corresponds to this string right here. I hope you can
 you can make sense of that. So the table is just put there as one long string and
 then in order to make the model realize you know what the table is you have
 these special embeddings. So usually in BERT you have what they're called
 position embeddings to indicate where in the sequence that is. So in a simpler in
 the simplest case these are embeddings for the numbers 0 1 2 3 4 and so on so
 wherever the position is. This you can all look up in the attention is all you
 need video I've made that if you are unfamiliar with transformer inputs. Then
 also the the segment embeddings simply indicate where what a token is part of. So
 for every token that's part of the query you see you have segment 0 embedding and
 for every token that's part of the table you have a segment 1 embedding. This is
 simply to tell the model hey this particular token is part of the question
 or part of the table. Then you have the new things so this paper newly
 introduces the following embeddings column and row embeddings. Now these for
 the question of course they don't make any sense but you have to put something
 here so you just put column 0 but for the table you see there is a column 1
 and column 2 and the this exactly so we've seen that this here is the header
 of column 1 and this is the header of column 2 and then it goes back column 1
 column 2 column 1 column 2 and you can see here this 0 is in column 1 and this
 one is in column 2 and this in column 1 again and the same for the rows so you
 have row 0 for the headers and then row 1 for the first two numbers and row 2
 for the second two numbers. So this is all of this so you see these two are in
 the first row and these two are in the second row. All of this is to tell the
 model all of this information down here is to tell the model how this table
 looks so if it wants to select the second column from the third row it
 would look in this information to see which cell to select and then the last
 thing they introduce is this so-called rank embeddings. Now as we've seen before
 if this first column here is maybe the sorry the number of days of something so
 this is the number of days and this second one is the number of reigns so
 how many championships the table can only be sorted at maximum by one of them
 so you want to sort of for each cell you want to tell the model let's extend that
 table by two numbers 4 and 1 so for each column you want to tell the model the
 ranking of the numbers so here it's pretty easy this is rank 1 this is rank
 2 this is rank 3 further on the left side this is rank 1 this is rank 2 down
 here and this is rank 3 so the model has an will have an if you give this
 information the model have an easier time to detect like give me the top 2 or
 something like this give me the worst give me the best give me the highest and
 so on the model will have an easier time doing that so that's why the rank here
 as you can see the zero and as also the number one are embedded rank one and the
 other two rank two because they're just lower now I don't feel I feel they could
 could have given a better example than this table I feel you could actually
 put real names here to make clearer not call one and call two and I feel you
 could give a somewhat smarter content because if you just look at the picture
 here you cannot see the correspondence of these rank tokens because in essence
 they are exactly equal as the row tokens but fortunately we can read the text oh
 there's the table ha so I have actually I've not seen that but I have discerned
 it correctly for this particular for this particular input alright I think
 that's the the half of the magic is how you encode the input in such a thing and
 this seems to be first of all a pretty cool idea but second of all it exactly
 is what this kind of new regime of NLP is about is that you basically put
 everything as a string you annotate it in a smart way and that lets the model
 figure out a lot of stuff about the input people used to people used to do
 the very different things so people if given a query and a table like this what
 people would do is they would somehow first of all get the table headers and
 and kind of guess the data types of the attributes and then they would formulate
 reformulate the query maybe also with a neural network maybe with something else
 into something like SQL in order to actually have an SQL statement to select
 the correct cells or perform the correct aggregations and that is somewhat
 brittle and it's just much less deep learning than this model so I like this
 part of the model now the problem of course is as we've seen in this multi
 step process so how do we first of all if you build if we want to build a cell
 selector that's pretty easy right we've seen this so we the cell selector is
 first column column selection and then second row selection and this can be
 multiple rows so that's fairly easy selecting cells either for just returning
 or for aggregation pretty easy but how do we do the actually the aggregation
 selection is also pretty easy because we can just do a multi class classifier
 right so the classifier will simply tell us a give us a distribution and then we
 see okay the sum aggregation is probably here the the what the model wants the
 real question is how do we train this and how this is trained is what I find
 really interesting so as we've seen they have training data the training data
 comes in the form of tables questions and answers as we've seen before we don't
 know how to get to those answers so when the question is which wrestler had the
 most number of rains we just know the answer is a Ric flair now they they do
 again a two-step process for their training data that mimics the two step
 process of the model so the first step is is the answer a number is the answer
 a number if no then it is definitely a cell selection task so they if it's not
 a number they just restrict themselves to selecting cells if the answer is not
 in the table then that just means that the correct thing is to select no cells
 and just say I can't answer this question if it is a number then again
 you have two options so is it in the table if yes we are in a weird situation
 if no not in table then it is an aggregation so if it is a number that is
 not in the table that means that the answer is a number there's not in the
 table that means the answer must be computed via one of these aggregations
 and if the answer is a number but is in the table then we are in this ambiguous
 answer setting where the it could be that we need to select the cell but it
 could also be that the same number by accident is in the table but actually
 needs to be computed from other numbers and they do this in the most deep
 blurny way possible is that they do basically a soft decision here so
 they let the model when they let it select what to compute they let it make
 a soft decision what do I mean by that so let's say you have these three
 operations count sum and average and you have the cell selection so the cell
 selector will basically tell you I will select three cells the three cells
 contain the number seven the number eight and the number three alright so
 and the question was I don't even know what the question was but the cell
 selector tells you these three cells are to be selected you do this by simply
 selecting the cells where the cell selector has a higher probability than
 one half now your your aggregation selection module gives you a softmax
 distribution over over the actions so it's not very much count here maybe
 that's 0.1 this here is maybe 0.3 and this is the 0.6 what you do is you
 simply compute all of them so you want to compute the count here which is three
 you want to compute the sum here which is 18 and then you want to compute the
 average which is six ha I made a good example by accident and then you simply
 weigh the outputs here by their probabilities so you say since the model
 wants point one puts what point one probability on the count I'm going to
 have 0.1 times 3 plus it wants point three times this so 0.3 times 18 plus
 0.6 times 6 now I'm not gonna so this is 6 plus point three plus 3.6 9.9 so that
 that's how the model computes things it simply puts probability on these
 operations here and then you simply take a weighted output with respect to the
 computation of all those things now I'm pretty sure that's completely invalid
 because for the same numbers for example the sum is going to have a much larger
 like variance than the average and and that's somewhat going the count maybe
 somewhere in between depending on the numbers so this just to take the weighted
 average here and then of course right so what they do is they do have this this
 is the model output and you have the correct answer let's say the correct
 answer was actually was to compute the the average so the correct answers six
 so what they do is simply they take the squared error and that's their loss
 actually they don't take the squared error they take a approximation to the
 squared error which is square until some Delta and then it's linear and this is
 simply to be a bit more outlier robust and they do other things to be more
 outlier robust but this so this is the model output and this is the correct
 answer and they simply count on the fact that this will this will back propagate
 so if you want to make these two things closer if you're the model right you
 have the option of simply putting more weight from the from the other ones on
 to the average operation and that will decrease the 9.9 because you as you can
 see both of these numbers will get smaller and no wait this isn't the yes
 sorry so you will you will decrease these numbers so this is the output we
 got from the weighted average right so if we decrease these weights you will
 put weight from here to here that will bring the number 9.9 down and that will
 get you closer to the answer you're looking for but you can also achieve
 this by you can achieve this even more right so this 9.9 is too high if we want
 to bring the 9.9 down we're much better off by taking some of that output and
 actually putting on this here because three is the lowest number right the
 only agreement here is that we want to take weight away from the 18 from the
 large one so I'm extremely surprised that this works given that it is so
 super ambiguous what the model should do with these operations and I I highly
 doubt that you can extend this so it's of course agnostic of what these
 aggregations are but to be able to extend this to many more aggregations
 is will I think lead to much more of these situations where the model is
 entirely unsure of where to put the mass of where to put the weight and I would be
 interested to see what happens if you have a data set with like 20 or 50 of
 these aggregations and not just three so this is the this is the let's say the
 the interesting part here the other if you go the other way when you have this
 cell selection task it is just to select a cell right and then you simply have
 the cell selector that part here that does the selection that you also you
 train every time simply to give each cell a weight right so this this is
 simply the softmax over column and then the softmax over rows and you can train
 that using the cross entropy now training this cell selector from data is
 pretty easy when it's a cell selection task right because the answer is in the
 table and or is not in the table and then you know to select no cell so you
 do have the training data that a particular cell is the correct cell and
 you can train the model to select that cell but it is actually a pretty hard
 task if it is for example you're looking for an average operation because not
 only do you are you not really sure that it's an average operation you just know
 that that kind of gives you the correct answer you also don't really know which
 cells to select for this average operation right because depending on
 which cells you select and of course that's going to be a soft selection as
 well the the average answer the average will be different depending on which
 cells you select so they're basically counting on this loss here to back
 propagate not only through the the selection of the aggregation to perform
 but also to the cell selector to set which cells to to select so from this
 weak signal it's almost like the reinforcement learning problem where you
 have the weak signal and you have like a billion ways to get your number closer
 to that signal and not not really accurate understanding of what you need
 to do is you're just relying on the model through lots and lots and lots and
 lots of data to kind of figure out which natural language questions to map to
 which cell selection and aggregation so this is it's a it seems like impossible
 but it works the last thing we need to talk about is this ambiguous answer
 setting and as you can imagine it's pretty simple that they also let the
 model do and a soft selection between the cell selection tasks so no
 aggregation and the aggregations to be performed and basically let the model
 figure out itself which one is better to do an aggregation or to do no
 aggregation suffice to say this this only works for pretty I think I think it
 only works for pretty limited amount of tasks pretty limited amount of questions
 and you might have spotted there even these questions that are follow-up
 questions which are another thing they build into the model and I don't I'm not
 really gonna talk about this but they do have this concept as well which I find
 maybe a bit out of place but maybe it's just part of their data set somewhere
 maybe it's just these companies want to get into this conversational mode so
 everything needs to be context dependent at the interesting part here is really
 the computation of the aggregates and specifically the question of which of
 these aggregations to choose and this again this is so surprising that it
 works and fairly fairly cool I think that is the gist of the paper they do
 extremely thorough evaluations here on these data sets and ablations to see
 what really counts and what doesn't I don't really want to go into that safe
 to say their results are better than anything else before I believe they I
 believe they're actually on par with another model but in one data set but
 they beat them on every other data set so that's you know that's cool I don't
 think there was a bar nevermind I invite you to check out this paper look for
 yourself they have the code online if you want to train a model like this
 yourself other than that thanks for listening if you like this content
 please subscribe like comment tell a friend and bye bye