Underspecification Presents Challenges for Credibility in Modern Machine Learning (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper google pipeline ml pipeline deep networks epidemiology theoretical underspecification overparameterization overfitting generalization out of distribution bert gender stereotypes distribution shift analysis performance bias correlation problems quality assurance
#ai #research #machinelearning

Deep Learning models are often overparameterized and have many degrees of freedom, which leads to many local minima that all perform equally well on the test set. But it turns out that even though they all generalize in-distribution, the performance of these models can be drastically different when tested out-of-distribution. Notably, in many cases, a good model can actually be found among all these candidates, but it seems impossible to select it. This paper describes this problem, which it calls underspecification, and gives several theoretical and practical examples.

OUTLINE:
0:00 - Into & Overview
2:00 - Underspecification of ML Pipelines
11:15 - Stress Tests
12:40 - Epidemiological Example
20:45 - Theoretical Model
26:55 - Example from Medical Genomics
34:00 - ImageNet-C Example
36:50 - BERT Models
56:55 - Conclusion & Comments

Paper: https://arxiv.org/abs/2011.03395

Abstract:
ML models often exhibit unexpectedly poor behavior when they are deployed in real-world domains. We identify underspecification as a key reason for these failures. An ML pipeline is underspecified when it can return many predictors with equivalently strong held-out performance in the training domain. Underspecification is common in modern ML pipelines, such as those based on deep learning. Predictors returned by underspecified pipelines are often treated as equivalent based on their training domain performance, but we show here that such predictors can behave very differently in deployment domains. This ambiguity can lead to instability and poor model behavior in practice, and is a distinct failure mode from previously identified issues arising from structural mismatch between training and deployment domains. We show that this problem appears in a wide variety of practical ML pipelines, using examples from computer vision, medical imaging, natural language processing, clinical risk prediction based on electronic health records, and medical genomics. Our results show the need to explicitly account for underspecification in modeling pipelines that are intended for real-world deployment in any domain.

Authors: Alexander D'Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D. Hoffman, Farhad Hormozdiari, Neil Houlsby, Shaobo Hou, Ghassen Jerfel, Alan Karthikesalingam, Mario Lucic, Yian Ma, Cory McLean, Diana Mincu, Akinori Mitani, Andrea Montanari, Zachary Nado, Vivek Natarajan, Christopher Nielson, Thomas F. Osborne, Rajiv Raman, Kim Ramasamy, Rory Sayres, Jessica Schrouff, Martin Seneviratne, Shannon Sequeira, Harini Suresh, Victor Veitch, Max Vladymyrov, Xuezhi Wang, Kellie Webster, Steve Yadlowsky, Taedong Yun, Xiaohua Zhai, D. Sculley

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi there! Today we'll look at Under Specification Presents Challenges for Credibility in Modern Machine Learning by Alexander Damour, Catherine Heller, Dan Moldovan and literally all of Google. All of Google is on this paper, including some others, including MIT and Google with a white space, but there is a lot of authors here and not sure what they all contributed, but the main authors are three main authors, which I guess is legit, but this more looks like some kind of a physics paper from CERN. But we'll dive into what the paper claims. It's sort of a paper that looks at a higher level onto machine learning pipelines, but gives very concrete examples for what it's talking about. So the problem that the paper identifies is this thing they call under specification, which is sort of related to problems we had in the past or that were identified in the past, but they make a clear distinction of what under specification is, to what problems it leads and how that manifests and also what the causes are to an extent. Well, it's a very long paper. I think it's some 30 pages long, the main text or so, so we won't go through all of it. I'll pick out some parts of where I think are relevant to the main story. I'll criticize it a bit because I think it warrants a bit of criticism and yeah, that's what we'll do. So bear with me. If you like videos like this, don't hesitate to share them out and tell your friends about it. Also let me know what you think in the comments. I think this is a good topic for discussing things. The question to keep in mind while going through this paper is, do they really demonstrate what they claim? So that was my kind of question when going through some of this. So let's actually just dive into the abstract. They say ML models often exhibit unexpectedly poor behavior when they are deployed in real world domains. I think we all get a sense of what that means and we all know of examples when ML models perform fine in our lab, in our training data and test data actually. But then when we deploy them into the world, they're not doing so fine. I say we identify under specification as a key reason for these failures. They're not saying it's the key reason, it's a key reason. So that's the important thing. Now they define it. They say an ML pipeline is under specified when it can return many predictors with equivalently strong held out performance in the training domain. Under specification is common in modern ML pipelines such as those based on deep learning. So I think this sentence isn't really complete here. So it's under specified when it can return many predictors with equivalently strong held out performance. So what that means is you have some sort of a test set, right? Big data set, sorry, train. You have a big training data set, you train your model on that and then you test it on a test set. And the training and the test set, they usually come from some sort of distribution. And what often happens is you simply split your data into a train and a test set. And with that you measure this some sort of generalization capability, right? So there are a number of assumptions here, namely that this is sort of an IID distributed data cloud. And the assumption is basically that the test data, the data to which your model will be applied in the real world, is sort of similar to the data you've trained it on. And if that is the case, then a procedure like this will give you a fairly good estimate of how your model is going to perform in practice. However, you then take that model and you deploy it to the real world. And the real world, look, I'm horrible at drawing real worlds, but in the real world, you might have, this is Europe, yay, Africa. In the real world, you might have very different distributions of data. And the model might not perform as well anymore. So this, of course, they're not the first ones to notice this particular problem, the fact that there's distribution shift and so on. What they are saying is that this procedure up here, let's say it's a deep learning system, there are many, many local minima of that deep learning system. So that starts from your choice of optimizer, your choice of batch size, hyperparameters, the choice of architecture of your network, and so on. So there are a number of hyperparameters, let's call them all hyperparameters, even like the different procedures and so on. So there are a number of hyperparameters, learning rate, architecture, batch size, all kinds of stuff. And what they experiment here with is the most innocuous of hyperparameters, which is the random seed. So even if everything else stays the same, and you switch up the random seed, you necessarily go into a different local minimum, right? All of these give you different models. We know that in deep learning, you have sort of a lot of local minima, actually, like you have a continuum of local minima. They are all as good as each other. And notably, so these are training models, notably, they all perform quite well on that test data set, right? So you train any of these models, maybe you switch up the random seed, and most of them will actually work quite well on the IID test data set. However, they will exhibit very, very different performance when you apply them to the real world. So maybe this model here, you apply it to the real world, and it works equally, it also works well. But maybe this model right here, you apply it to the real world, it all of a sudden doesn't work. So the under specification problem that they identify is when all the models work well, all the models from your training procedure work equally well on the test set. However, they perform very differently in the real world, namely, there would actually be a at least one model like this one here, that does perform well even in the real world. However, there is another one, at least one other that doesn't perform well like this. So the pipeline is underspecified. This train test split simply doesn't capture the variation that some important property of the real world. So the pipeline that produces the model is doesn't care about that feature. So it's pretty much random, whether or not that feature will be included or excluded or important or not important. And it's pretty much depends on which local minima you happen to be in. And just by looking at the test set, you can't differentiate whether or not that model will perform well in the real world, or not. This is under specification. It's very different from the usual domain shift argument. Usually you say, well, the test set simply isn't the same as the real world. And therefore, the model performs well on the test set, but then in the real world, not so much right here, it's more specific, you say, there would be one of these good models that we get out of this procedure, one of the random seeds would actually work well in the real world. However, another one doesn't. So of course, that is a problem. And they, so the the way they go about the paper is they say, they give some examples of how that is. And in my opinion, the examples don't really convince me like I see their point. However, the examples are, let's say half convincing. And then at the end, they they give some recommendations for I mean, there is some work in this. Namely, what you have to do is you have to add constraints, right? If you want to solve this problem, there's two ways either you can test models, you can take all of the models that come out of your pipeline, test each one of them on the real world on the things you care about. And the one that works, you know, you deploy that. However, it means that you then again need some kind of test data set from that real world. The other way is to actually, since the model is under specified, try to bring in more specifications that you care about during the training pipeline, making sure that this model that you care about is the one that actually turns out to be returned. They don't demonstrate this here. So this is my criticism, they don't, they don't, they demonstrate the problem. I think they demonstrate the problem in a way that doesn't convince me. They also do not demonstrate a solution. So they don't ever go ahead and say, now we actually perform this additional specification and look, what turns out is still a good performing model, but with that thing fixed, they don't do that. Yeah, so that's keep an eye out for that. So we'll go, as I said, through the paper. But first, a bit more of the abstract. So you just hear it in their words, they say predictors returned by under specified pipelines are often treated as equivalent based on their training domain performance. But we show that there that that such predictors can behave very differently in deployment domains. This ambiguity can lead to instability in poor model behavior and practice, and is a distinct failure mode from previously identified issues from arising from structural mismatch between training and deployment domains. So that's what I said, it's, it's a different problem than the classic domain shift or data drift or whatever you might want to call it. We show that this problem appears in a wide variety of practical and mental pipelines using examples from computer vision, medical imaging, yada, yada, yada. Our results show that the need to explicitly account for under specification in modeling pipelines that are intended for real world deployment in any domain. I mean, yeah, fair enough. This is actually a problem, right? And you, you, if you deploy ML in the real world, you would be, you know, it, it's very appropriate to actually care about these types of problems. I'm not saying you shouldn't care about this. Yeah, so let's go to, let's go to actually jump in the first example. So they have this notion of what they call a stress test. Okay. So a stress test is, as I understand it is nothing else than you test whether or not you test like one particular aspect of the model. So they're going to have a couple of examples. One example, they have an NLP pipeline where you're supposed to infer, I don't know, do pronoun resolution. And the stress test, one of the stress tests would be whether or not that model is sensitive to gender stereotypes. Okay. So the, the, the assumption is kind of pronoun resolution should be like just linguistic thing. It shouldn't really have any bias towards any gender stereotypes and whatnot, or maybe not overly so if you compare it to actual world biases. And the stress test would be, let's measure that particular dimension. So this, this gender stereotype dimension in the model and see how that performs. So that's the stress test. And what we are specifically looking for is, is there a large variance? So is there models that behave the same on the training and the test set, but have a large variance in these stress tests. So the first model here is this epidemiological model. So they say a simple epidemiological model, which appropriate for our times, I guess, specifies how disease how infectious disease moves through a population, given certain parameters, right? So there are two parameters, you can see the differential equations right here. There are two parameters, namely, there is this beta right here represents the transmission rate of the disease from the infected to susceptible populations. And the parameter D, which is this thing here, represents the average duration that an infected individual remains infectious. So once you plug in those parameters, and you start with like some pieces of some, some initial population, I guess, the susceptible population, this S is susceptible, I is infected, and R is recovered. So you start with 100% susceptible. And then you let this and zero infected zero recovered, you let this play out, and you see how well that works. So this is a model. And it will give you curves like this, okay. So you can see depending on the D parameter and the beta parameter, you have different curves like this, they all sort of look like this. So here is number of infected at the beginning, it's zero. And then of course, like it shoots up. And but then as kind of herd immunity, I guess kicks in, this goes down again. So it's a quite a simple model. And what their goal is here, they say, look, let's say, just hypothetically, hypothetically, this is the beginning of a pandemic, just making this up. And I give you some data points, right? So at the beginning, we're at zero, then we have some, then some more, then some more. Now please predict the trajectory of the of this epidemic from these data points. So what you want to do is you want to fit these two parameters to the data points, there is actually a unique solution. However, because of the exponential rise of the trajectory, the unique the solution is numerically not well specified. Okay, so they say importantly, during the early stages of an epidemic, when the observations are small, the parameters of the model are under specified by this training task. This is because at this stage, the number of susceptible is approximately constant at the at the total population size as the total at the total population. So that means if you have low number of infected people, the amount of people that could get infected is still like pretty much everyone, there is no no type of herd immunity yet. And the number of infections grows approximately exponentially at this rate. So you can see that approximately, approximately what you're dealing with is this rate right here. And you can see both parameters are in this rate. So if you derive some number for this, let's say this you derive from your data points that this must be five, this is the rate at which the exponential curve grows, there are many settings of beta and D that make this number five, right? In fact, there are infinitely many pairs that make this number be five. So they say this is a classic example of under specification, okay, there are many different predictors, each of which returns a good predictor on the data that you have. And you can actually you could split this into train and test, you could split these data points, you can say, I'll take three data points as a train and one as a test. And still, there would be many, many predictors that are fit the data here you see two of them. So the blue and the red, they fit the data equally well, right here. However, they have obviously very different trajectories. So they say this is an example of under specification. And here already, like I have agree. I mean, yes, yes, if you do it like this numerically, these look kind of similar, but it's like clearly one fits more than the other, right. So I'm not sure that that is is a good example for this under specification. But we can you know, we can give you can give kind of the the benefit here and say, okay, they want to give a simple model. So this is one of these models where it's under specified. So it performs well on this data. But then if you look at this data, it performs drastically differently, right? That's that's the important part here is drastically different. So if the real trajectory of the of the epidemic is something like this, then there is there is a predictor, namely d equal 28, that actually performs well, right? It's not that that training setup is different from the real world. It's that the variance of predictors is so large with respect to the data over here, that there might be some that perform well, but the others perform pretty, pretty poorly. And they say this is not only this is not only the case for you know, this initial fit. But if you do the same, and you simply use a different initialization, so you different simply use a different initialization for your parameters, namely, you either use a gamma or a normal distribution, that will already turn out to give you very different results. So here depends on where it was initialized, and different initialization distribution result in different distribution of predicted trajectories. So this is much more I feel an example of what they want to demonstrate. So here, depending on how you initialize the model, the resulting model that it tends to give you right, they do many different runs right here. And you can clearly see that the blue curves that were initialized with a normal distribution are in general kind of on average, significantly lower than the red curves, right? Same data, same procedure, same everything. But you get in expectation, even different outcomes simply by how you initialize the parameters. This is I feel this is a very good example, right here of what they want to say, not so much the early training data. But you get the point that that they say the under specification leaves this variance okay. Now, what would a good specification look like? So in this case, a good specification, a good would either be that you somehow know you somehow have a theoretical reason for choosing one of these two initializers, this could one specification be that could solve the problem. Another one that is probably more practical one would simply be to incorporate data from over here. And thereby you, you know, which model you should pick, which in an epidemic, it's not really it's like, well, I can tell you how it turns out once I know how it turns out, right? Yeah, so and that that's a bit of a problem, because it already shows you sometimes adding these more specifications or checking, checking whether or not the model does what you want it to do in this specific axis that has a large variance is just not possible, like here. But the example is, you know, it's the example. So the next thing they do is they analyze this in a theoretical model. So they have this theoretical model right here. This is kind of a two layer neural network, where the first layer is completely random. Okay, this is a random this is not trained, what's trained is this thing right here. So it's sort of kind of a linear model, it's a it's sort of a model of a neural network that people often use in theoretical analysis, you assume some kind of distribution on the data, then you assume some kind of distribution on the weight matrix on the weight matrix entries. And then all you do is you train the theta parameter right here. And you can make some theoretical statements about what happens with that model. So their goal here is to show that their their goal is to show the following. What is obviously let's let's say we keep the same data, okay, we keep the same data distribution or the same data. We sample this W right here. Now we can imagine W one, W two, W three, these are all different weight matrices, okay. So can we come up with a model that performs well on all the weight matrices that we would kind of throw at it. But that doesn't. But if we if we just plug in kind of different data, it doesn't it stops performing well in one particular axis, right. So as long as we as long as we only look at the training distribution, we're fine. But then there is this one particular axis that the model just fails for some weight matrices, but not for others. Okay, so that's that's going to be the theoretical goal here is to construct as closely as possible, a model that conforms to the to the claims right here. So what they do is they make use of adversarial perturbations, where they say, we can construct, we construct a weight matrix. Where is it? We construct a weight matrix here, for any given weight matrix, a shift can be chosen such that it has a small norm, so that it's essentially the same data that goes into the model. To it leaves the risk of an independently sampled W mostly unchanged, which is exactly what we you know, what we have specified is that if I simply evaluate if I train the model, and I simply evaluate it on my original data, then everything's fine. Okay. But it drastically increases the risk of W zero. So what it says is that if I have such a model like I have above, then I can construct a situation where I pick, I simply pick one weight matrix, say this one right here, I can derive a data set x zero, or x, let's call that x three for W three, I can derive a data set x three, such that all the other weight matrices will work just fine on that data set, right, they will work the same as my original data right here, everything's fine. However, this particular one won't work on that data set. And that is going to that is going to result from an adversarial perturbation targeted at exactly that. So this, this thing here constructs a data set that is according to their own claims. Okay, so it's a cool thing to show that this is possible. If you have an over specified model, you can generally do you can generally construct a situation that exactly conforms to their claims. However, I, I, this is cool in theory, but I don't think they demonstrate this too much in the real examples right here. So yeah, just just, maybe this was unclear, I'm not the best at explaining this, this type of stuff. But what you can imagine is that the weight matrices that you get out of your training procedure, they can be fairly different, right, let's just call them vectors. So this is w one, this is w two, w three, w four, if your neural network just had two, two different weights, so the weight matrices can be drastically different, and the solutions to them can be drastically different, but I can construct kind of an adversarial data set that is, let's say, exactly into the this is going to very simplified exactly into the let's say, opposite direction of one particular weight matrix, so that it will work just fine with this weight matrix, it will work just fine with this with this, because you have kind of the projection onto them is well specified. But if I try to project it onto this one, maybe I should have drawn it exactly orthogonal. But you get what I mean, I can sort of target one of these models. And then by definition, that one particular model that is as good as all the other models on the regular data will fail for this particular data set, whereas all the other models will still work just fine. It's kind of a theoretical analysis by construction. Yeah, cool. But, you know, if you make a claim, and then you construct a situation that exactly conforms to your claims, then of course, it's going to conform to your claims. Yeah, so this is more according to the real world. So this is a medical genomics example, where you can see the training, the training data, they have training data, they have evaluation data that comes from the same distribution, and then they have evaluation data that comes out of distribution. So this is more like a domain drift domain shift example. Okay. And our question is going to be how do these things relate? So you can see that if you train on the training data, and then you evaluate on the training data, you get this is mean squared normalized mean squared error, so lower is better, you get kind of a variance of models. So these are all the models that kind of come out of the training procedure. And the red dot is a specific heuristic that that performs just a bit better. This is actually it's so what it does is you have a bunch of data points, but the data points sort of form clusters. And what these methods do is they take one representative out of each cluster, like so one representative, and then they train a model just on the representative data on the representatives. And that's supposed to just because these data points are all very correlated, if they're in the same cluster, that kind of gives a better performance, the red dot simply is a very special heuristic to choose that representative, whereas the blue dots here simply choose these representatives at random. So you can conceivably say that all these models, like the difference is simply how these representatives are selected. And you can see they all turn out fairly similar with the red dot being just a little bit better. If you go to the test set on the same data, you can see the performance drops. But you know, still, everything performs like pretty well, the range of performance here is fairly small. So all of these models, you would say they perform pretty okay, ish. But now you go to the set set, say out of distribution data, and the range of performance is just very, very big. And the point here I think they're trying to make is that look at the best performing models right here, look at them, they are on the level of the performance of your models in the test data set in the in distribution test data set. However, not all of them, right. So the good performing model would be in the models that you get, but you simply can't tell from just looking at the test data set. And that that is, according to their claim. And they have a further graphic right here where they show look, it's not it's not as easy as saying the let's just take the best one here, because that's going to be the best one here. So here a plot, they compare how well a model does, and the eval set in distribution versus the eval set out of distribution. And you can see, the correlation is if it's there, it's fairly weak. So you like you would expect some line like this, if that was just stretched out, right? If this thing was just stretched, you would expect like a line. But here, there's just no way to tell for this particular data set. Okay, so that's, that's an example of what they mean by under specification. However, I, like I, I fail to see, like, I see that these low points right here are kind of on the level of the test distribution. But I am not like, I fail to see what the difference is to a classic data drift, just because they are on the on the same level. Right? I, I don't think it's that different. Like here, the mean performance simply drops and the variance between the models increases. And if I had a different eval set, the ordering would be different, and it would look the same, but the ordering of models would be different and so on. What you'd have to do to for me, like you, I wonder, for example, is it the case in this step as well? So what here what here, if you did the same analysis, would it turn out that what performs well in the training data set also performs well in the test data set? Or is it also pretty, pretty random from the training data set to predict the at least the order of tests at performance? They never do anything like this. If this is substantially different here, then you can make an argument. Well, this is a different thing than simply some sort of generalization. This is really kind of due to this under specification, because going from this data set to this data set, you sort of have a different spec. But to me, it seems that this is just kind of a domain drift problem. And if you look closely, actually, the performance right here is lower than the best performance here, right? So that this technically does not fall under their definition if you go strictly. So I'm not really sure what to make of these sort of examples. I get what they're trying to say. But it seems to me that except for the theoretical thing where they construct the examples, it doesn't convince me that it's not just domain drift, okay? Like it's not just the same problem that other people have described. And secondly, it also doesn't convince me that adding the specification will solve the problem because in the experiment so far, notice we have never seen a method from them to say, let's just fix the problem. Let's add the specification. And then we show that we can really keep this performance, right? The key thing is you want to keep this performance, but you want to bring this performance up, right? So far, we've had these kind of fundamental trade offs. And these have often arisen, let's say explainability or fairness and so on, or actually domain adaptation is, if you want to bring this down, a natural effect is going to be to bring this up. So, you know, even if there are good models right here, it might be that to in order to reach those models, you actually have to weaken the training procedure in order to consistently reach those models. This is not demonstrated in the paper that this is even possible. Okay, so they have a bunch of more case studies. For example, they have this kind of ImageNet C example, where ImageNet C kind of takes ImageNet and applies a bunch of random but let's say, well specified perturbations on it. And again, they show the same thing right here. They show that look, all these models, they perform relatively equally on the just plain test set of ImageNet, but the span of these models, they are trained all the same, just the random seed is different, right? And they have a huge span of performance on these individual things. And what you'll notice also here or here is that it's not always the same model. So the model that is good at the pixelate thing will be not so good at the contrast thing and so on. So the question is going to be, which the paper also doesn't solve, is going to be that, you know, these kind of stress tests, they are in very, very specific things like pixelate, I can think of a million perturbations to images that are kind of orthogonal to pixelate, it is going to be very impossible to specify all of them, right to remove this under specifications. So the question is, is probably by adding the specification of pixelate, you simply worsen the problem for any of the other things that you have still not specified, plus you probably worsen a little bit your performance on the actual test set if you incorporate that into training. So the paper still hasn't shown that that is even even possible. What is interesting is, yeah, here, they basically say you cannot predict the performance on one of these perturbations from the others. So they appear to be completely orthogonal. So it's not just enough to have a bunch of perturbations and then kind of be confident that the model is sort of robust to all the perturbations. I think the core message of the paper is that if you care about a specific axis, you have to go and check for that specific axis, right? Otherwise, you don't know what your model is doing. It could be doing something good, but it could be doing something bad if you don't specifically care about it. They do the same thing with kind of these skin lesions. So they have all kinds of demonstration here. In NLP, they do tests with BERT. And this is interesting because not only do they test different seeds for fine tuning BERT, but they also test different seeds for pre training. So in these language models, you have like a pre training phase, and then you have a fine tuning phase, and both of them have kind of random seeds. So they are going to show that even let's say the random seed of the pre training will actually already play a big role in how these models perform in these stress tests. I find this to be pretty interesting. So they do this with respect to these gender datasets, which have been constructed to sort of assess fairness of these models. And so what you're going to have is data like the following. So you're going to have the sentence, let's say a doctor is walking. So it's always going to be like some sort of profession, okay, used in a sentence. And then what you do is you simply replace that entity with a man or a woman, right, you replace it twice. And you ask your model, you perform, you embed all of these sentences, and then you ask your model how similar are those sentences, I presume by simply taking the inner product of the of the embeddings, or you can actually train it. Okay, so they say part of glue, our ensemble of predictors achieve consistent accuracy, measure in terms of correlation with human provided similarity scores ranging from this to that. Okay, so you have kind of a model that can predict similarity in text, just similarity, it has, it does not, it knows nothing about gender, right, you simply train it on a data set to predict similarity in text. And then you ask it, so this sentence that I have here, this reference sentence, is it more similar to when I replace the entity with a woman, or is it more similar to when I replace the entity with a man? Okay, and what you look at is the the difference between the two. So if this is a positive, this is a positive number, that means that the sentence is more similar to when you replace it with the word woman. And when you have a negative number, the same for men. And if the model is, let's say insensitive to the gender dimension, then you expect a difference here of zero, at least in expectation, right. So a model that does not learn a gender correlation for a given profession will have an expected similarity delta of zero. We are particularly interested in the extent to which the similarity delta for each profession correlates with the percentage of women actually employed in that profession, as measured by US Bureau of Labor Statistics. Right, this is, in my opinion, this is already an improved assessment from what usually happens in these, in these fairness literature things where they just say, well, if it's anything but 5050, we are angry, which I get, I get it if you, you know, some cases, you need to build a model that is actually 5050. But if, if you want to assess things like they assess here, like the question, the question is, does the model spurious ly pick up this thing? So if the model does something like if the model is, let's say, perfect, and does only the task we needed to do, it will learn the association between a profession and the gender in the exact proportion that it kind of happens in the text, which I guess is proportional to the proportionate which is happens in the world. If, however, the model for some reason uses this thing as a feature more or less than it should, then we see a discrepancy. And why is that important that it's important because if we then deploy this model, right, we simply take so the model here is going to be the axis here is going to be 00. And the model can perfectly solve the task by simply being here, right, it's actually best to be here, where this delta between the similarity and the profession percentage is zero. But the model can probably solve the task equally well by being here, or here, or here, or here, right, it can solve the task equally well. However, if we just happen to pick at the end, we pick one model, if we happen to pick this model right here, that model just by more or less chance, has a much higher association with one gender to particular professions than the other. And depending on what we use the model for, like we seldomly use the model on the exact task and data that we trained it on, depending on what we use it for, this might cause some some adverse effects. Okay, so I want to stress that this is not the same as your kind of classic fairness literature, this really considers all these models, they perform like equally well on the test set of that particular task. And since it's overspecified or underspecified, overparameterized, there are many, many ways to solve tasks, some of these ways will include this feature, some of these ways will include actually the opposite feature. And if we kind of pick one that's at the extreme, then the model is going to have that feature. And that might not that might not be important for this task. But it might cause something bad for a task that we ultimately apply it on. So they do this similarity and they do pronoun resolution. And so they come up with different things, they say there is a large spread in correlation with BLS statistics. On the STS test correlations range from point three to point seven. On the pronoun resolution task, the range is this. As a point of comparison prior work on gender short, pronoun resolution found correlation ranging for that. Okay, so we are in the in the same ball ballpark as prior work. They say there is a weak relationship between test accuracy, performance and gendered correlation. So there is a Spearman correlation coefficient for of point oh eight, which is a weak correlation, right? In fact, the confidence interval includes zero. Oh, that's for pronoun resolution. So for for the for the similarity, it's point two one, which is an okay correlation, the confidence interval just barely includes zero. So we're fairly sure. I'm not a statistician, don't grill me about p values. This they say this indicates that learning accurate predictors does not require learning strong gendered correlations, which is a statement you can make though, I would say such a over over parameterized under specified model will probably pick up this feature here fairly often since the correlation is there, right? But they are right, it does not require it does not require strong correlations. Okay. And they say, third, the encoding of spurious correlation is sensitive to the random seed at pre training, and not just fine tuning. So this is very interesting, especially in the pronoun resolution tasks, the pronoun resolution test, don't want to go into it too much here. But so here you can see two different runs, so two different random seeds that result in two very different. So here is the similarity delta, this is this this minus this we observed before plotted against this percentage female by occupation for individual occupations. And you can see here, this predictor has a stronger correlation than this predictor right here. Now I've thought about it, I'm still not sure which one is let's say, let's call it the better one. Because I'm, yeah, I'm not sure like because that you can say the bottom predictor has less of a correlation with actual occupation. I think that makes it worse. Right. But you might argue that a model just shouldn't depend, or shouldn't care. But then the delta is not zero. Whereas this top predictor actually has the zero here at fairly at the point where it's 5050. So I'm going to tacitly argue that the top predictor is the one you want. But I don't know. The important part of the paper doesn't make a strong opinionated claim about which one you want. The paper actually just says, you should be aware that both predictors solve the task very well. However, one they're drastically different in how they treat this feature. So here you can see, there's not really a correlation between this score and the test set accuracy, you can't tell from the test set. What you know, can tell from the test set how it's going to perform in this particular stress test. And this is very interesting in the pronoun resolution task, they here they plot by different pre training seats, and you can see they clearly cluster, right. So even the pre training seed has an influence later on this on this performance. I guess it's kind of logical, but it's still interesting to see that this clusters so well, while all these things solve the task. Same so that it basically means that you can't just take like a bird checkpoint and then fine tune it with like an objective in there that you might already be worried about how the pre training happened. I guess maybe you can fix it. I know that's what they don't show. So they analyze it a bit more. They say they take 20 of those predictors here to better understand the differences between predictors in our example, we analyze the structure in how similarity scores produced by the predictors in our ensemble deviate from the ensemble mean. Here we find that the main axis of variation aligns at least in its at its extremes, with differences in how predictors represent stereotypical associations between profession and gender. So these data sets, by the way, they are annotated, you know, they are constructed such that the kind of stereotypes manifest or don't manifest depending on how much your model has picked those up during training. Specifically, we perform principal component analysis over similarity score produced by 20 fine tunings of a single bird checkpoint. So 20 different models. We plot the first principal component, which contains 22% of the variation in score deviations against the female participation percentages in figure nine. Notably examples in the region where the first principal components values are strongly negative, include some of the strongest gender imbalances. So let's look at this graphic right here, because this this is where I kind of sort of get skeptical. Okay, so let's understand these plots on the left right here. So what you have is the first principal component of this kind of of this resulting similarity scores. So I'm going to guess each of these dots here is one of these models. So you can see, and I'm going to guess that each of these line is like one of these professions. Okay, so for a given profession like this, this here appears to be a profession where let's say approximately that has a 20% female participate occupation rate. And the spread here is going to be how the different models happen to to manifest in the first principal component. So the first principal component, you know, the axis of largest variation in the data set. So the first thing that is very notable here is that these models are spread out quite a bit, right? So they are they are they perform like sometimes it's very the it's very negative. Sometimes it's very positive for the same thing, right? This is what is strange. Or this is a thing that this paper points out all these models perform equally well on the test set of the task that they care about. However, so this here is when you put men as a subject. So up here, the 100, these occupations that are listed here would be something like I don't know, mine, mine worker, oil rig worker or something like this. And on the bottom, you'd have kind of the more stereotypical female professions like nurse or something like this. So a couple of things to note here. So what they what they do is the red dots here are theirs. So they say, we'll take the extremes and extremes are just whenever I think this is here is negative one. So they take the extremes and they look at them here and they kind of make a point of the first principle component in its extremes displays kind of the most most anti stereotypical examples. Okay, so it you have to see here is these dots are where the first principle component is loaded negatively by a lot. And the sentences these are the red dot sentences right there red dots. Those are those sentences. A receptionist is crawling is the sentence and the plot is for men as a subject. So this is the when measured when you measure the similarity between a receptionist is crawling and a man is crawling. You ask how similar are those sentences compared to when I enter a woman is crawling. Sorry, compared to the similarity of a receptionist is crawling with a woman is crawling. Right. So this is the data. This is fairly it's fairly meta, right. So their claim is that this first principle component kind of incorporates this feature by a lot. And I think their their point is kind of see even when we don't train this stuff, there are models that that very much rely on kind of these or that very much over rely on these kind of stereotypes. However, that this is very, I feel it's it's a bit it's a bit shady because I mean, look at look at this data, right, you can't like you can't just pick these outliers like here. These are outliers too. And even if you look here, like they conveniently pick. So I guess they conveniently pick such that these things here are left out, you can see here, it's woman as a subject. So what you'd expect here, if this is really the models pick up a lot of these kind of spurious correlation, what you'd expect is a line like this, right, you have like shift here and then up here because you know, 100% women like the first component will load a lot. You don't see that at all. Right. And here you see a little bit you see a little bit a slope like this. But I don't think that just and especially if you look at the noise between the things like this is here. And then this is over here. Right. So like the in between noise is way bigger. To go and claim you had the first principle components contain something like this and then we don't look at these outliers up here. I, I don't know. Yeah, so this this doesn't seem to me like, I see what they're trying to say. And what is concerning is that there is such a big spread among the models, right? Within this professions, there is a giant spread. These are the same performing models. So I see the what they're trying to say, but I don't think the point they're making here. I don't know if this is politics or something that they have to kind of bring in these these types of topics. But you know, they also look at with respect to others and they show look, these models perform differently with respect to different stress test dimensions and notably the ordering isn't the same. But again, I feel that this is simply this might be just a problem of domain shift rather than what they're claiming. And lastly, they have kind of a a test on these other stress tests that are also NLP stress tests. And you can see that the models perform quite differently. So there's a spread right here. Within each of these, the red bar is the spread on the actual test set, as I understand it. And then these are the different pre training seeds. And you can again see that even the pre training seed will have a big effect right here. So yeah, again, what I would like to see is kind of how does the even does even the training performance predict the test performance on the same distribution that would already be quite informative. As you can see right here, you can't really predict one of the stress tests from the other. The question is just can you even do this for the training to the test set because that would inform you whether or not this is a property of this stress test being in a different direction, one direction that you didn't capture. If if these stress tests are really meant to show that look, you can't really tell this axis that you didn't specify this is really because of under specification, you would expect that from the training performance, you could at least predict the test performance somewhat or from the test performance you could predict on an ID ID test set. I'm going to assume that it is somewhat like this, but I'm also not sure that you like that this is anything to rely on. And the last thing they do is kind of a lab study where they have kind of vital signals and they predict whether or not there is a medical problem. And again, you can see here they even test different architectures and so on. And what they're basically the point is the point is the same. But it's just shown in a different data. It's pretty cool that they have lots of different different examples right here, but I don't want to go into the lab thing. So their discussion at the end, I think is kind of kind of weak because I mean, what they say is our findings underscore the need to thoroughly test models on application specific tasks, and in particular to check that the performance on these tasks is stable. I mean, I fully agree with that, right? If you if you deploy your model into some sort of real world application, please test whether it actually works in that real world application. But it seems to me that that is not it's not a solution fully to the problem because as we saw in the epidemiology paper, that sometimes just isn't possible. And also, you know, it is the case that not everyone can train a language model. So we kind of need pre trained checkpoints. Maybe the goal is that we provide like maybe Google, if they provide one birth checkpoint, let's say they provide 50, right, and then people can go ahead and check which one actually is good or bad on on their particular dimension that they care about that maybe the pre training didn't care about. That would, I think that would be a practical solution to the problem. If you can't specify it. And what I would say also is that it's not clear to me that it is always possible, even, you know, in theory, maybe, but it is not clear to me that it is always possible to add the specification that you want, and keep the same performance, I see that there are predictors in the set that they consider that have that. But that doesn't mean that once you add the constraint, the training procedure reaches that same performance, and specifically keeps the performance on the test set. So that's kind of a number of criticisms on this paper. All in all, I mean, it's, it's a paper that you generally can agree with, right, can agree with the sentiment, and also the analysis, the examples are, of course, real. And the problem is real. And, yeah, especially for a company like Google, this is fairly important because they build big models and deploy big models. All right, let me know what you think about this. I'll see you next time. Bye bye.
 Hi there! Today we'll look at Under Specification Presents Challenges for
 Credibility in Modern Machine Learning by Alexander Damour, Catherine Heller,
 Dan Moldovan and literally all of Google. All of Google is on this paper,
 including some others, including MIT and Google with a white space, but there is a
 lot of authors here and not sure what they all contributed, but the main
 authors are three main authors, which I guess is legit, but this more looks
 like some kind of a physics paper from CERN. But we'll dive into what the paper
 claims. It's sort of a paper that looks at a higher level onto machine learning
 pipelines, but gives very concrete examples for what it's talking about. So
 the problem that the paper identifies is this thing they call under
 specification, which is sort of related to problems we had in the past or that
 were identified in the past, but they make a clear distinction of what under
 specification is, to what problems it leads and how that manifests and also
 what the causes are to an extent. Well, it's a very long paper. I think it's some
 30 pages long, the main text or so, so we won't go through all of it. I'll pick out
 some parts of where I think are relevant to the main story. I'll criticize it a
 bit because I think it warrants a bit of criticism and yeah, that's what we'll do.
 So bear with me. If you like videos like this, don't hesitate to share them
 out and tell your friends about it. Also let me know what you think in the
 comments. I think this is a good topic for discussing things. The
 question to keep in mind while going through this paper is, do they really
 demonstrate what they claim? So that was my kind of question when
 going through some of this. So let's actually just dive into the
 abstract. They say ML models often exhibit unexpectedly poor behavior when
 they are deployed in real world domains. I think we all get a sense of
 what that means and we all know of examples when ML models perform fine in
 our lab, in our training data and test data actually. But then when we deploy
 them into the world, they're not doing so fine. I say we identify under
 specification as a key reason for these failures. They're not saying it's the key
 reason, it's a key reason. So that's the important thing. Now they define it. They
 say an ML pipeline is under specified when it can return many predictors with
 equivalently strong held out performance in the training domain. Under
 specification is common in modern ML pipelines such as those based on deep
 learning. So I think this sentence isn't really complete here. So it's
 under specified when it can return many predictors with equivalently strong
 held out performance. So what that means is you have some sort of a test set,
 right? Big data set, sorry, train. You have a big training data set, you train your
 model on that and then you test it on a test set. And the training and the test
 set, they usually come from some sort of distribution. And what often happens is
 you simply split your data into a train and a test set. And with that you measure
 this some sort of generalization capability, right? So there are a number
 of assumptions here, namely that this is sort of an IID distributed data
 cloud. And the assumption is basically that the test data, the data to which
 your model will be applied in the real world, is sort of similar to the data
 you've trained it on. And if that is the case, then a procedure like this will
 give you a fairly good estimate of how your model is going to perform in
 practice. However, you then take that model and you deploy it to the real
 world. And the real world, look, I'm horrible at drawing real worlds, but in
 the real world, you might have, this is Europe, yay, Africa. In the real world,
 you might have very different distributions of data. And the model
 might not perform as well anymore. So this, of course, they're not the first
 ones to notice this particular problem, the fact that there's distribution shift
 and so on. What they are saying is that this procedure up here, let's say it's a
 deep learning system, there are many, many local minima of that deep learning
 system. So that starts from your choice of optimizer, your choice of batch size,
 hyperparameters, the choice of architecture of your network, and so on.
 So there are a number of hyperparameters, let's call them all hyperparameters,
 even like the different procedures and so on. So there are a number of
 hyperparameters, learning rate, architecture, batch size, all kinds of
 stuff. And what they experiment here with is the most innocuous of
 hyperparameters, which is the random seed. So even if everything else stays
 the same, and you switch up the random seed, you necessarily go into a
 different local minimum, right? All of these give you different models. We know
 that in deep learning, you have sort of a lot of local minima, actually, like you
 have a continuum of local minima. They are all as good as each other. And
 notably, so these are training models, notably, they all perform quite well on
 that test data set, right? So you train any of these models, maybe you switch up
 the random seed, and most of them will actually work quite well on the IID test
 data set. However, they will exhibit very, very different performance when you
 apply them to the real world. So maybe this model here, you apply it to the
 real world, and it works equally, it also works well. But maybe this model right
 here, you apply it to the real world, it all of a sudden doesn't work. So the
 under specification problem that they identify is when all the models work
 well, all the models from your training procedure work equally well on the test
 set. However, they perform very differently in the real world, namely,
 there would actually be a at least one model like this one here, that does
 perform well even in the real world. However, there is another one, at least
 one other that doesn't perform well like this. So the pipeline is
 underspecified. This train test split simply doesn't capture the variation
 that some important property of the real world. So the pipeline that produces the
 model is doesn't care about that feature. So it's pretty much random, whether or
 not that feature will be included or excluded or important or not important.
 And it's pretty much depends on which local minima you happen to be in. And
 just by looking at the test set, you can't differentiate whether or not that
 model will perform well in the real world, or not. This is under
 specification. It's very different from the usual domain shift argument.
 Usually you say, well, the test set simply isn't the same as the real world.
 And therefore, the model performs well on the test set, but then in the real
 world, not so much right here, it's more specific, you say, there would be one of
 these good models that we get out of this procedure, one of the random seeds
 would actually work well in the real world. However, another one doesn't. So
 of course, that is a problem. And they, so the the way they go about the paper
 is they say, they give some examples of how that is. And in my opinion, the
 examples don't really convince me like I see their point. However, the examples
 are, let's say half convincing. And then at the end, they they give some
 recommendations for I mean, there is some work in this. Namely, what you have
 to do is you have to add constraints, right? If you want to solve this problem,
 there's two ways either you can test models, you can take all of the models
 that come out of your pipeline, test each one of them on the real world on
 the things you care about. And the one that works, you know, you deploy that.
 However, it means that you then again need some kind of test data set from
 that real world. The other way is to actually, since the model is under
 specified, try to bring in more specifications that you care about
 during the training pipeline, making sure that this model that you care about is
 the one that actually turns out to be returned. They don't demonstrate this
 here. So this is my criticism, they don't, they don't, they demonstrate the
 problem. I think they demonstrate the problem in a way that doesn't convince
 me. They also do not demonstrate a solution. So they don't ever go ahead and
 say, now we actually perform this additional specification and look, what
 turns out is still a good performing model, but with that thing fixed, they
 don't do that. Yeah, so that's keep an eye out for that. So we'll go, as I said,
 through the paper. But first, a bit more of the abstract. So you just hear it
 in their words, they say predictors returned by under specified pipelines
 are often treated as equivalent based on their training domain performance. But
 we show that there that that such predictors can behave very differently
 in deployment domains. This ambiguity can lead to instability in poor model
 behavior and practice, and is a distinct failure mode from previously identified
 issues from arising from structural mismatch between training and deployment
 domains. So that's what I said, it's, it's a different problem than the
 classic domain shift or data drift or whatever you might want to call it. We
 show that this problem appears in a wide variety of practical and mental
 pipelines using examples from computer vision, medical imaging, yada, yada, yada.
 Our results show that the need to explicitly account for under
 specification in modeling pipelines that are intended for real world deployment
 in any domain. I mean, yeah, fair enough. This is actually a problem, right? And
 you, you, if you deploy ML in the real world, you would be, you know, it, it's
 very appropriate to actually care about these types of problems. I'm not saying
 you shouldn't care about this. Yeah, so let's go to, let's go to actually jump
 in the first example. So they have this notion of what they call a stress test.
 Okay. So a stress test is, as I understand it is nothing else than you
 test whether or not you test like one particular aspect of the model. So
 they're going to have a couple of examples. One example, they have an NLP
 pipeline where you're supposed to infer, I don't know, do pronoun resolution. And
 the stress test, one of the stress tests would be whether or not that model is
 sensitive to gender stereotypes. Okay. So the, the, the assumption is kind of
 pronoun resolution should be like just linguistic thing. It shouldn't really have
 any bias towards any gender stereotypes and whatnot, or maybe not overly so if
 you compare it to actual world biases. And the stress test would be, let's
 measure that particular dimension. So this, this gender stereotype dimension in
 the model and see how that performs. So that's the stress test. And what we are
 specifically looking for is, is there a large variance? So is there models that
 behave the same on the training and the test set, but have a large variance in
 these stress tests. So the first model here is this epidemiological model. So
 they say a simple epidemiological model, which appropriate for our times, I guess,
 specifies how disease how infectious disease moves through a population,
 given certain parameters, right? So there are two parameters, you can see the
 differential equations right here. There are two parameters, namely, there is
 this beta right here represents the transmission rate of the disease from
 the infected to susceptible populations. And the parameter D, which is this thing
 here, represents the average duration that an infected individual remains
 infectious. So once you plug in those parameters, and you start with like some
 pieces of some, some initial population, I guess, the susceptible population, this
 S is susceptible, I is infected, and R is recovered. So you start with 100%
 susceptible. And then you let this and zero infected zero recovered, you let
 this play out, and you see how well that works. So this is a model. And it will
 give you curves like this, okay. So you can see depending on the D parameter and
 the beta parameter, you have different curves like this, they all sort of look
 like this. So here is number of infected at the beginning, it's zero. And then of
 course, like it shoots up. And but then as kind of herd immunity, I guess kicks
 in, this goes down again. So it's a quite a simple model. And what their goal is
 here, they say, look, let's say, just hypothetically, hypothetically, this is
 the beginning of a pandemic, just making this up. And I give you some data points,
 right? So at the beginning, we're at zero, then we have some, then some more, then
 some more. Now please predict the trajectory of the of this epidemic from
 these data points. So what you want to do is you want to fit these two parameters
 to the data points, there is actually a unique solution. However, because of the
 exponential rise of the trajectory, the unique the solution is numerically not
 well specified. Okay, so they say importantly, during the early stages of
 an epidemic, when the observations are small, the parameters of the model are
 under specified by this training task. This is because at this stage, the number
 of susceptible is approximately constant at the at the total population size as
 the total at the total population. So that means if you have low number of
 infected people, the amount of people that could get infected is still like
 pretty much everyone, there is no no type of herd immunity yet. And the number of
 infections grows approximately exponentially at this rate. So you can
 see that approximately, approximately what you're dealing with is this rate
 right here. And you can see both parameters are in this rate. So if you
 derive some number for this, let's say this you derive from your data points
 that this must be five, this is the rate at which the exponential curve grows,
 there are many settings of beta and D that make this number five, right? In
 fact, there are infinitely many pairs that make this number be five. So they
 say this is a classic example of under specification, okay, there are many
 different predictors, each of which returns a good predictor on the data
 that you have. And you can actually you could split this into train and test,
 you could split these data points, you can say, I'll take three data points as
 a train and one as a test. And still, there would be many, many predictors
 that are fit the data here you see two of them. So the blue and the red, they
 fit the data equally well, right here. However, they have obviously very
 different trajectories. So they say this is an example of under specification.
 And here already, like I have agree. I mean, yes, yes, if you do it like this
 numerically, these look kind of similar, but it's like clearly one fits more
 than the other, right. So I'm not sure that that is is a good example for this
 under specification. But we can you know, we can give you can give kind of the
 the benefit here and say, okay, they want to give a simple model. So this is one of
 these models where it's under specified. So it performs well on this data. But
 then if you look at this data, it performs drastically differently, right?
 That's that's the important part here is drastically different. So if the real
 trajectory of the of the epidemic is something like this, then there is
 there is a predictor, namely d equal 28, that actually performs well, right? It's
 not that that training setup is different from the real world. It's that the
 variance of predictors is so large with respect to the data over here, that there
 might be some that perform well, but the others perform pretty, pretty poorly. And
 they say this is not only this is not only the case for you know, this initial
 fit. But if you do the same, and you simply use a different initialization, so
 you different simply use a different initialization for your parameters,
 namely, you either use a gamma or a normal distribution, that will already
 turn out to give you very different results. So here depends on where it was
 initialized, and different initialization distribution result in different
 distribution of predicted trajectories. So this is much more I feel an example of
 what they want to demonstrate. So here, depending on how you initialize the
 model, the resulting model that it tends to give you right, they do many different
 runs right here. And you can clearly see that the blue curves that were
 initialized with a normal distribution are in general kind of on average,
 significantly lower than the red curves, right? Same data, same procedure, same
 everything. But you get in expectation, even different outcomes simply by how
 you initialize the parameters. This is I feel this is a very good example, right
 here of what they want to say, not so much the early training data. But you
 get the point that that they say the under specification leaves this variance
 okay. Now, what would a good specification look like? So in this case, a good
 specification, a good would either be that you somehow know you somehow have a
 theoretical reason for choosing one of these two initializers, this could one
 specification be that could solve the problem. Another one that is probably
 more practical one would simply be to incorporate data from over here. And
 thereby you, you know, which model you should pick, which in an epidemic, it's
 not really it's like, well, I can tell you how it turns out once I know how it
 turns out, right? Yeah, so and that that's a bit of a problem, because it
 already shows you sometimes adding these more specifications or checking,
 checking whether or not the model does what you want it to do in this specific
 axis that has a large variance is just not possible, like here. But the example
 is, you know, it's the example. So the next thing they do is they analyze this
 in a theoretical model. So they have this theoretical model right here. This is
 kind of a two layer neural network, where the first layer is completely random.
 Okay, this is a random this is not trained, what's trained is this thing
 right here. So it's sort of kind of a linear model, it's a it's sort of a
 model of a neural network that people often use in theoretical analysis, you
 assume some kind of distribution on the data, then you assume some kind of
 distribution on the weight matrix on the weight matrix entries. And then all you
 do is you train the theta parameter right here. And you can make some
 theoretical statements about what happens with that model. So their goal
 here is to show that their their goal is to show the following. What is
 obviously let's let's say we keep the same data, okay, we keep the same data
 distribution or the same data. We sample this W right here. Now we can imagine
 W one, W two, W three, these are all different weight matrices, okay. So can
 we come up with a model that performs well on all the weight matrices that we
 would kind of throw at it. But that doesn't. But if we if we just plug in
 kind of different data, it doesn't it stops performing well in one particular
 axis, right. So as long as we as long as we only look at the training
 distribution, we're fine. But then there is this one particular axis that the
 model just fails for some weight matrices, but not for others. Okay, so
 that's that's going to be the theoretical goal here is to construct as
 closely as possible, a model that conforms to the to the claims right here.
 So what they do is they make use of adversarial perturbations, where they
 say, we can construct, we construct a weight matrix. Where is it? We construct
 a weight matrix here, for any given weight matrix, a shift can be chosen
 such that it has a small norm, so that it's essentially the same data that goes
 into the model. To it leaves the risk of an independently sampled W mostly
 unchanged, which is exactly what we you know, what we have specified is that if
 I simply evaluate if I train the model, and I simply evaluate it on my original
 data, then everything's fine. Okay. But it drastically increases the risk of W
 zero. So what it says is that if I have such a model like I have above, then I
 can construct a situation where I pick, I simply pick one weight matrix, say this
 one right here, I can derive a data set x zero, or x, let's call that x three for
 W three, I can derive a data set x three, such that all the other weight matrices
 will work just fine on that data set, right, they will work the same as my
 original data right here, everything's fine. However, this particular one won't
 work on that data set. And that is going to that is going to result from an
 adversarial perturbation targeted at exactly that. So this, this thing here
 constructs a data set that is according to their own claims. Okay, so it's a cool
 thing to show that this is possible. If you have an over specified model, you can
 generally do you can generally construct a situation that exactly conforms to
 their claims. However, I, I, this is cool in theory, but I don't think they
 demonstrate this too much in the real examples right here. So yeah, just just,
 maybe this was unclear, I'm not the best at explaining this, this type of stuff.
 But what you can imagine is that the weight matrices that you get out of your
 training procedure, they can be fairly different, right, let's just call them
 vectors. So this is w one, this is w two, w three, w four, if your neural network
 just had two, two different weights, so the weight matrices can be drastically
 different, and the solutions to them can be drastically different, but I can
 construct kind of an adversarial data set that is, let's say, exactly into the
 this is going to very simplified exactly into the let's say, opposite direction of
 one particular weight matrix, so that it will work just fine with this weight
 matrix, it will work just fine with this with this, because you have kind of the
 projection onto them is well specified. But if I try to project it onto this one,
 maybe I should have drawn it exactly orthogonal. But you get what I mean, I can
 sort of target one of these models. And then by definition, that one particular
 model that is as good as all the other models on the regular data will fail for
 this particular data set, whereas all the other models will still work just fine.
 It's kind of a theoretical analysis by construction. Yeah, cool. But, you know,
 if you make a claim, and then you construct a situation that exactly
 conforms to your claims, then of course, it's going to conform to your claims.
 Yeah, so this is more according to the real world. So this is a medical genomics
 example, where you can see the training, the training data, they have training
 data, they have evaluation data that comes from the same distribution, and
 then they have evaluation data that comes out of distribution. So this is
 more like a domain drift domain shift example. Okay. And our question is going
 to be how do these things relate? So you can see that if you train on the
 training data, and then you evaluate on the training data, you get this is mean
 squared normalized mean squared error, so lower is better, you get kind of a
 variance of models. So these are all the models that kind of come out of the
 training procedure. And the red dot is a specific heuristic that that performs
 just a bit better. This is actually it's so what it does is you have a bunch of
 data points, but the data points sort of form clusters. And what these methods do
 is they take one representative out of each cluster, like so one
 representative, and then they train a model just on the representative data
 on the representatives. And that's supposed to just because these data
 points are all very correlated, if they're in the same cluster, that kind
 of gives a better performance, the red dot simply is a very special heuristic
 to choose that representative, whereas the blue dots here simply choose these
 representatives at random. So you can conceivably say that all these models,
 like the difference is simply how these representatives are selected. And you
 can see they all turn out fairly similar with the red dot being just a little bit
 better. If you go to the test set on the same data, you can see the performance
 drops. But you know, still, everything performs like pretty well, the range of
 performance here is fairly small. So all of these models, you would say they
 perform pretty okay, ish. But now you go to the set set, say out of distribution
 data, and the range of performance is just very, very big. And the point here I
 think they're trying to make is that look at the best performing models right
 here, look at them, they are on the level of the performance of your models in the
 test data set in the in distribution test data set. However, not all of them,
 right. So the good performing model would be in the models that you get, but you
 simply can't tell from just looking at the test data set. And that that is,
 according to their claim. And they have a further graphic right here where they
 show look, it's not it's not as easy as saying the let's just take the best one
 here, because that's going to be the best one here. So here a plot, they compare
 how well a model does, and the eval set in distribution versus the eval set out
 of distribution. And you can see, the correlation is if it's there, it's fairly
 weak. So you like you would expect some line like this, if that was just
 stretched out, right? If this thing was just stretched, you would expect like a
 line. But here, there's just no way to tell for this particular data set. Okay,
 so that's, that's an example of what they mean by under specification. However, I,
 like I, I fail to see, like, I see that these low points right here are kind of
 on the level of the test distribution. But I am not like, I fail to see what the
 difference is to a classic data drift, just because they are on the on the same
 level. Right? I, I don't think it's that different. Like here, the mean
 performance simply drops and the variance between the models increases.
 And if I had a different eval set, the ordering would be different, and it would
 look the same, but the ordering of models would be different and so on. What you'd
 have to do to for me, like you, I wonder, for example, is it the case in this step
 as well? So what here what here, if you did the same analysis, would it turn out
 that what performs well in the training data set also performs well in the test
 data set? Or is it also pretty, pretty random from the training data set to
 predict the at least the order of tests at performance? They never do anything
 like this. If this is substantially different here, then you can make an
 argument. Well, this is a different thing than simply some sort of
 generalization. This is really kind of due to this under specification, because
 going from this data set to this data set, you sort of have a different spec.
 But to me, it seems that this is just kind of a domain drift problem. And if
 you look closely, actually, the performance right here is lower than the
 best performance here, right? So that this technically does not fall under
 their definition if you go strictly. So I'm not really sure what to make of
 these sort of examples. I get what they're trying to say. But it seems to me
 that except for the theoretical thing where they construct the examples, it
 doesn't convince me that it's not just domain drift, okay? Like it's not just
 the same problem that other people have described. And secondly, it also doesn't
 convince me that adding the specification will solve the problem because in the
 experiment so far, notice we have never seen a method from them to say, let's
 just fix the problem. Let's add the specification. And then we show that we
 can really keep this performance, right? The key thing is you want to keep this
 performance, but you want to bring this performance up, right? So far, we've had
 these kind of fundamental trade offs. And these have often arisen, let's say
 explainability or fairness and so on, or actually domain adaptation is, if you
 want to bring this down, a natural effect is going to be to bring this up. So, you
 know, even if there are good models right here, it might be that to in order to
 reach those models, you actually have to weaken the training procedure in order
 to consistently reach those models. This is not demonstrated in the paper that
 this is even possible. Okay, so they have a bunch of more case studies. For
 example, they have this kind of ImageNet C example, where ImageNet C kind of
 takes ImageNet and applies a bunch of random but let's say, well specified
 perturbations on it. And again, they show the same thing right here. They show
 that look, all these models, they perform relatively equally on the just plain
 test set of ImageNet, but the span of these models, they are trained all the
 same, just the random seed is different, right? And they have a huge span of
 performance on these individual things. And what you'll notice also here or here
 is that it's not always the same model. So the model that is good at the pixelate
 thing will be not so good at the contrast thing and so on. So the question is
 going to be, which the paper also doesn't solve, is going to be that, you
 know, these kind of stress tests, they are in very, very specific things like
 pixelate, I can think of a million perturbations to images that are kind of
 orthogonal to pixelate, it is going to be very impossible to specify all of
 them, right to remove this under specifications. So the question is, is
 probably by adding the specification of pixelate, you simply worsen the problem
 for any of the other things that you have still not specified, plus you
 probably worsen a little bit your performance on the actual test set if
 you incorporate that into training. So the paper still hasn't shown that that
 is even even possible. What is interesting is, yeah, here, they basically
 say you cannot predict the performance on one of these perturbations from the
 others. So they appear to be completely orthogonal. So it's not just enough to
 have a bunch of perturbations and then kind of be confident that the model is
 sort of robust to all the perturbations. I think the core message of the paper
 is that if you care about a specific axis, you have to go and check for that
 specific axis, right? Otherwise, you don't know what your model is doing. It
 could be doing something good, but it could be doing something bad if you
 don't specifically care about it. They do the same thing with kind of these skin
 lesions. So they have all kinds of demonstration here. In NLP, they do tests
 with BERT. And this is interesting because not only do they test different
 seeds for fine tuning BERT, but they also test different seeds for pre training. So
 in these language models, you have like a pre training phase, and then you have a
 fine tuning phase, and both of them have kind of random seeds. So they are going
 to show that even let's say the random seed of the pre training will actually
 already play a big role in how these models perform in these stress tests. I
 find this to be pretty interesting. So they do this with respect to these gender
 datasets, which have been constructed to sort of assess fairness of these models.
 And so what you're going to have is data like the following. So you're going to
 have the sentence, let's say a doctor is walking. So it's always going to be
 like some sort of profession, okay, used in a sentence. And then what you do is
 you simply replace that entity with a man or a woman, right, you replace it
 twice. And you ask your model, you perform, you embed all of these sentences,
 and then you ask your model how similar are those sentences, I presume by simply
 taking the inner product of the of the embeddings, or you can actually train it.
 Okay, so they say part of glue, our ensemble of predictors achieve
 consistent accuracy, measure in terms of correlation with human provided
 similarity scores ranging from this to that. Okay, so you have kind of a model
 that can predict similarity in text, just similarity, it has, it does not, it
 knows nothing about gender, right, you simply train it on a data set to predict
 similarity in text. And then you ask it, so this sentence that I have here, this
 reference sentence, is it more similar to when I replace the entity with a woman,
 or is it more similar to when I replace the entity with a man? Okay, and what you
 look at is the the difference between the two. So if this is a positive, this is a
 positive number, that means that the sentence is more similar to when you
 replace it with the word woman. And when you have a negative number, the same for
 men. And if the model is, let's say insensitive to the gender dimension, then
 you expect a difference here of zero, at least in expectation, right. So a model
 that does not learn a gender correlation for a given profession will have an
 expected similarity delta of zero. We are particularly interested in the extent to
 which the similarity delta for each profession correlates with the percentage
 of women actually employed in that profession, as measured by US Bureau of
 Labor Statistics. Right, this is, in my opinion, this is already an improved
 assessment from what usually happens in these, in these fairness literature
 things where they just say, well, if it's anything but 5050, we are angry, which I
 get, I get it if you, you know, some cases, you need to build a model that is
 actually 5050. But if, if you want to assess things like they assess here,
 like the question, the question is, does the model spurious ly pick up this thing?
 So if the model does something like if the model is, let's say, perfect, and does
 only the task we needed to do, it will learn the association between a
 profession and the gender in the exact proportion that it kind of happens in the
 text, which I guess is proportional to the proportionate which is happens in the
 world. If, however, the model for some reason uses this thing as a feature more
 or less than it should, then we see a discrepancy. And why is that important
 that it's important because if we then deploy this model, right, we simply take
 so the model here is going to be the axis here is going to be 00. And the model
 can perfectly solve the task by simply being here, right, it's actually best to
 be here, where this delta between the similarity and the profession percentage
 is zero. But the model can probably solve the task equally well by being here, or
 here, or here, or here, right, it can solve the task equally well. However, if
 we just happen to pick at the end, we pick one model, if we happen to pick this
 model right here, that model just by more or less chance, has a much higher
 association with one gender to particular professions than the other. And
 depending on what we use the model for, like we seldomly use the model on the
 exact task and data that we trained it on, depending on what we use it for, this
 might cause some some adverse effects. Okay, so I want to stress that this is
 not the same as your kind of classic fairness literature, this really
 considers all these models, they perform like equally well on the test set of
 that particular task. And since it's overspecified or underspecified,
 overparameterized, there are many, many ways to solve tasks, some of these ways
 will include this feature, some of these ways will include actually the opposite
 feature. And if we kind of pick one that's at the extreme, then the model is
 going to have that feature. And that might not that might not be important
 for this task. But it might cause something bad for a task that we
 ultimately apply it on. So they do this similarity and they do pronoun
 resolution. And so they come up with different things, they say there is a
 large spread in correlation with BLS statistics. On the STS test correlations
 range from point three to point seven. On the pronoun resolution task, the range
 is this. As a point of comparison prior work on gender short, pronoun resolution
 found correlation ranging for that. Okay, so we are in the in the same ball
 ballpark as prior work. They say there is a weak relationship between test
 accuracy, performance and gendered correlation. So there is a Spearman
 correlation coefficient for of point oh eight, which is a weak correlation,
 right? In fact, the confidence interval includes zero. Oh, that's for pronoun
 resolution. So for for the for the similarity, it's point two one, which is
 an okay correlation, the confidence interval just barely includes zero. So
 we're fairly sure. I'm not a statistician, don't grill me about p values.
 This they say this indicates that learning accurate predictors does not
 require learning strong gendered correlations, which is a statement you
 can make though, I would say such a over over parameterized under specified
 model will probably pick up this feature here fairly often since the
 correlation is there, right? But they are right, it does not require it does not
 require strong correlations. Okay. And they say, third, the encoding of spurious
 correlation is sensitive to the random seed at pre training, and not just fine
 tuning. So this is very interesting, especially in the pronoun resolution
 tasks, the pronoun resolution test, don't want to go into it too much here. But
 so here you can see two different runs, so two different random seeds that result
 in two very different. So here is the similarity delta, this is this this minus
 this we observed before plotted against this percentage female by occupation for
 individual occupations. And you can see here, this predictor has a stronger
 correlation than this predictor right here. Now I've thought about it, I'm still
 not sure which one is let's say, let's call it the better one. Because I'm, yeah,
 I'm not sure like because that you can say the bottom predictor has less of a
 correlation with actual occupation. I think that makes it worse. Right. But you
 might argue that a model just shouldn't depend, or shouldn't care. But then the
 delta is not zero. Whereas this top predictor actually has the zero here at
 fairly at the point where it's 5050. So I'm going to tacitly argue that the top
 predictor is the one you want. But I don't know. The important part of the
 paper doesn't make a strong opinionated claim about which one you want. The paper
 actually just says, you should be aware that both predictors solve the task very
 well. However, one they're drastically different in how they treat this feature.
 So here you can see, there's not really a correlation between this score and the
 test set accuracy, you can't tell from the test set. What you know, can tell from
 the test set how it's going to perform in this particular stress test. And this is
 very interesting in the pronoun resolution task, they here they plot by
 different pre training seats, and you can see they clearly cluster, right. So even
 the pre training seed has an influence later on this on this performance. I guess
 it's kind of logical, but it's still interesting to see that this clusters so
 well, while all these things solve the task. Same so that it basically means
 that you can't just take like a bird checkpoint and then fine tune it with
 like an objective in there that you might already be worried about how the
 pre training happened. I guess maybe you can fix it. I know that's what they
 don't show. So they analyze it a bit more. They say they take 20 of those
 predictors here to better understand the differences between predictors in our
 example, we analyze the structure in how similarity scores produced by the
 predictors in our ensemble deviate from the ensemble mean. Here we find that the
 main axis of variation aligns at least in its at its extremes, with differences in
 how predictors represent stereotypical associations between profession and
 gender. So these data sets, by the way, they are annotated, you know, they are
 constructed such that the kind of stereotypes manifest or don't manifest
 depending on how much your model has picked those up during training.
 Specifically, we perform principal component analysis over similarity
 score produced by 20 fine tunings of a single bird checkpoint. So 20 different
 models. We plot the first principal component, which contains 22% of the
 variation in score deviations against the female participation percentages in
 figure nine. Notably examples in the region where the first principal components
 values are strongly negative, include some of the strongest gender imbalances.
 So let's look at this graphic right here, because this this is where I kind of
 sort of get skeptical. Okay, so let's understand these plots on the left right
 here. So what you have is the first principal component of this kind of of
 this resulting similarity scores. So I'm going to guess each of these dots here
 is one of these models. So you can see, and I'm going to guess that each of
 these line is like one of these professions. Okay, so for a given
 profession like this, this here appears to be a profession where let's say
 approximately that has a 20% female participate occupation rate. And the
 spread here is going to be how the different models happen to to manifest in
 the first principal component. So the first principal component, you know, the
 axis of largest variation in the data set. So the first thing that is very
 notable here is that these models are spread out quite a bit, right? So they
 are they are they perform like sometimes it's very the it's very negative.
 Sometimes it's very positive for the same thing, right? This is what is
 strange. Or this is a thing that this paper points out all these models
 perform equally well on the test set of the task that they care about. However,
 so this here is when you put men as a subject. So up here, the 100, these
 occupations that are listed here would be something like I don't know, mine,
 mine worker, oil rig worker or something like this. And on the bottom, you'd have
 kind of the more stereotypical female professions like nurse or something like
 this. So a couple of things to note here. So what they what they do is the red
 dots here are theirs. So they say, we'll take the extremes and extremes are just
 whenever I think this is here is negative one. So they take the extremes
 and they look at them here and they kind of make a point of the first principle
 component in its extremes displays kind of the most most anti stereotypical
 examples. Okay, so it you have to see here is these dots are where the first
 principle component is loaded negatively by a lot. And the sentences these are
 the red dot sentences right there red dots. Those are those sentences. A
 receptionist is crawling is the sentence and the plot is for men as a subject. So
 this is the when measured when you measure the similarity between a
 receptionist is crawling and a man is crawling. You ask how similar are those
 sentences compared to when I enter a woman is crawling. Sorry, compared to the
 similarity of a receptionist is crawling with a woman is crawling. Right. So this
 is the data. This is fairly it's fairly meta, right. So their claim is that this
 first principle component kind of incorporates this feature by a lot. And I
 think their their point is kind of see even when we don't train this stuff,
 there are models that that very much rely on kind of these or that very much
 over rely on these kind of stereotypes. However, that this is very, I feel it's
 it's a bit it's a bit shady because I mean, look at look at this data, right,
 you can't like you can't just pick these outliers like here. These are outliers
 too. And even if you look here, like they conveniently pick. So I guess they
 conveniently pick such that these things here are left out, you can see here,
 it's woman as a subject. So what you'd expect here, if this is really the
 models pick up a lot of these kind of spurious correlation, what you'd expect
 is a line like this, right, you have like shift here and then up here because you
 know, 100% women like the first component will load a lot. You don't see
 that at all. Right. And here you see a little bit you see a little bit a slope
 like this. But I don't think that just and especially if you look at the noise
 between the things like this is here. And then this is over here. Right. So like
 the in between noise is way bigger. To go and claim you had the first principle
 components contain something like this and then we don't look at these outliers
 up here. I, I don't know. Yeah, so this this doesn't seem to me like, I see what
 they're trying to say. And what is concerning is that there is such a big
 spread among the models, right? Within this professions, there is a giant spread.
 These are the same performing models. So I see the what they're trying to say, but
 I don't think the point they're making here. I don't know if this is politics or
 something that they have to kind of bring in these these types of topics. But
 you know, they also look at with respect to others and they show look, these
 models perform differently with respect to different stress test dimensions and
 notably the ordering isn't the same. But again, I feel that this is simply this
 might be just a problem of domain shift rather than what they're claiming. And
 lastly, they have kind of a a test on these other stress tests that are also
 NLP stress tests. And you can see that the models perform quite differently. So
 there's a spread right here. Within each of these, the red bar is the spread on
 the actual test set, as I understand it. And then these are the different pre
 training seeds. And you can again see that even the pre training seed will have
 a big effect right here. So yeah, again, what I would like to see is kind of how
 does the even does even the training performance predict the test performance
 on the same distribution that would already be quite informative. As you can
 see right here, you can't really predict one of the stress tests from the other.
 The question is just can you even do this for the training to the test set
 because that would inform you whether or not this is a property of this stress
 test being in a different direction, one direction that you didn't capture. If
 if these stress tests are really meant to show that look, you can't really tell
 this axis that you didn't specify this is really because of under specification,
 you would expect that from the training performance, you could at least predict
 the test performance somewhat or from the test performance you could predict on an
 ID ID test set. I'm going to assume that it is somewhat like this, but I'm also
 not sure that you like that this is anything to rely on. And the last thing
 they do is kind of a lab study where they have kind of vital signals and they
 predict whether or not there is a medical problem. And again, you can see
 here they even test different architectures and so on. And what they're
 basically the point is the point is the same. But it's just shown in a different
 data. It's pretty cool that they have lots of different different examples
 right here, but I don't want to go into the lab thing. So their discussion at
 the end, I think is kind of kind of weak because I mean, what they say is our
 findings underscore the need to thoroughly test models on application
 specific tasks, and in particular to check that the performance on these tasks
 is stable. I mean, I fully agree with that, right? If you if you deploy your
 model into some sort of real world application, please test whether it
 actually works in that real world application. But it seems to me that that
 is not it's not a solution fully to the problem because as we saw in the
 epidemiology paper, that sometimes just isn't possible. And also, you know, it is
 the case that not everyone can train a language model. So we kind of need
 pre trained checkpoints. Maybe the goal is that we provide like maybe Google,
 if they provide one birth checkpoint, let's say they provide 50, right, and
 then people can go ahead and check which one actually is good or bad on on their
 particular dimension that they care about that maybe the pre training didn't
 care about. That would, I think that would be a practical solution to the
 problem. If you can't specify it. And what I would say also is that it's not
 clear to me that it is always possible, even, you know, in theory, maybe, but
 it is not clear to me that it is always possible to add the specification that
 you want, and keep the same performance, I see that there are predictors in the
 set that they consider that have that. But that doesn't mean that once you add
 the constraint, the training procedure reaches that same performance, and
 specifically keeps the performance on the test set. So that's kind of a number
 of criticisms on this paper. All in all, I mean, it's, it's a paper that you
 generally can agree with, right, can agree with the sentiment, and also the
 analysis, the examples are, of course, real. And the problem is real. And,
 yeah, especially for a company like Google, this is fairly important because
 they build big models and deploy big models. All right, let me know what you
 think about this. I'll see you next time. Bye bye.