How far can we scale up? Deep Learning's Diminishing Returns (Article Review)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper scale co2 gpt-3 bert language models environment large scale large language models deep neural networks transformers imagenet datasets language modeling training cost openai microsoft google google ai facebook research transfer learning meta learning exponential scale overparameterization
#deeplearning #co2 #cost

Deep Learning has achieved impressive results in the last years, not least due to the massive increases in computational power and data that has gone into these models. Scaling up currently promises to be a reliable way to create more performant systems, but how far can we go? This article explores the limits of exponential scaling in AI, and what people are doing to get around this problem

OUTLINE:
0:00 - Intro & Overview
1:00 - Deep Learning at its limits
3:10 - The cost of overparameterization
5:40 - Extrapolating power usage and CO2 emissions
10:45 - We cannot just continue scaling up
13:25 - Current solution attempts
15:25 - Aside: ImageNet V2
17:50 - Are symbolic methods the way out?

Paper: https://spectrum.ieee.org/deep-learning-computational-cost

Image by Ralf Vetterle from Pixabay: https://pixabay.com/images/id-1752876/

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi there, I saw this article in IEEE spectrum called deep learnings diminishing returns, the cost of improvement is becoming unsustainable. This is by Nielse Thompson, Christian Greenwald, Kihon Lee, and Gabrielle F. Monso. And I thought it was an interesting read, because it talks about the computational limits that we're reaching with deep learning today. And I have it over here in an annotatable form, though it might not look as pretty. I think the article, it leads up to the point where it shows just how much compute will be needed to make further improvements in deep learning and what the consequences of that might be, and some of the ways that people are trying to get around it. Now, I don't agree with everything the article says, but I think it's a it's a pretty neat read. It's pretty short. So I thought we can talk about it a little bit. So the article starts out with essentially praising deep learning for achieving so many things, for example, translating between languages, predicting how proteins fold, and many other things playing games as complex as go. They say it has risen relatively recently, but it has a long history. They mentioned 1958. And Frank Rosenblatt at Cornell, they designed the first artificial neural network, they say Rosenblatt's ambitions outpaced the capability of his era. And he knew it. Apparently, he said, as the number of connections in the network increases, the burden of a conventional digital computer soon becomes excessive. So why are deep neural networks working? Because of course, computers have increased in power massively, just for computing power, there has been whatever a 10 million fold increase, according to Moore's law. And that's usually just measured in something like CPU instructions. And now we went even beyond that building special purpose hardware such as GPUs, which aren't actually special purpose for this, but also TPUs. So they say these more powerful computers have made it possible to construct networks with vastly more connections and neurons enhance greater ability to model complex phenomena. And of course, these are the deep neural networks that power most of today's advances in AI, they draw a comparison right here, they say like Rosenblatt before them, today's deep learning researchers are nearing the frontier of what their tools can achieve, essentially claiming that we are in a similar situation today, we have the models that can achieve things, and we know pretty much that scaling them up can increase performance. However, we're kind of at the limits of how much we can scale. For example, I reported on this that Sam Altman apparently said GPT four will not be much bigger than GPT three, it will be trained more efficiently, we'll have some smartness in it on how it's processed, it will use more compute, but it will not necessarily be that much bigger in scale. So the first thing the article touches about deep learning is the fact that deep networks are over parameterized. For example, the noisy student model has some 480 million parameters, yet is trained on only 1.2 million labeled images, which is the image net data set. Now, of course, the noisy student model, if I understand correctly, also may leverage unlabeled data. But granted, today's neural networks are massively over parameterized, they have more parameters than data points available, therefore, they should horribly overfit. But they don't. They say classically, this would lead to overfitting where the model not only learns general trends, but also the random vagaries of the data it was trained on. Deep learning avoids this trap by initializing the parameters randomly, and then iteratively adjusting sets of them to better fit the data using a method called stochastic gradient descent. Surprisingly, this procedure has been proven to ensure that the learned model generalizes well. Now, I'm pretty sure that we are not yet sure why exactly deep networks don't overfit or why they generalize as they get over parameterized. I know there are some proofs around SGD and so on. But these proofs usually require assumptions that just make them completely lose touch to reality. But the core message is true, deep networks are over parameterized. And that is probably one of the reasons why they work so well. And being over parameterized, they are quite flexible. They say at the good news is that deep learning provides enormous flexibility. The bad news is that this flexibility comes at an enormous computational cost. This unfortunate reality has two parts. They say the first part is true of all statistical models to improve performance by factor of k, at least k squared more data points must be used to train the model. Does this really hold for all statistical models? Is this from the same theory that says the statistical models should overfit when they're over parameterized? I'm not sure. The second part, they say, of the computational cost comes explicitly from over parameterization. Once accounted for, this yields a total computational cost for improvement of at least k to the fourth power, meaning for a tenfold improvement, you would need to increase the computation by 10,000. Now, regardless of whether you think the theoretical analysis is actually accurate here, again, this is from the same area that says these models should overfit horribly, it doesn't matter, because these people have actually collected data. And they say theory tells us that computing needs to scale with at least the fourth power of the improvement in performance. In practice, the actual requirements have scaled with at least the ninth power. So when you actually measure how much people need to scale computation in order to achieve a given performance, then it's actually it's much worse than the theory predicts. In fact, they have these neat graphs right here. So on the left, you can see the percent error, I believe this is the image net classification data set. And on this axis, you can see the time. Now here you can see that over time, as time progresses, the error has come down and down and down again, as new state of the art models were proposed ever since the 2012 success of Alex net. And if you extrapolate that you can pretty clearly see that around 2025, we should be at approximately 5% of error. See, I thought you'd had to actually do something to reach a new state of the art on image net. But as it turns out, we just need to sit here and wait until 2025. Okay, jokes aside, they overlay this graph with another graph right here. And that is the comparison of again, percent error on the y axis. But now it's not the year in which the achievement was made. But it is number of computations in billions of flops. And notice the log scale down here. Now I have to say this graph right here makes it pretty clear that there might be something like a relationship, even maybe a linear relationship that you can extrapolate right here. I'm not so sure like these models are up here and then goes like here and then it goes here and then it goes here. And then it goes over here to 2020. And really without that, you probably have a line that goes something like this. Now in any case, if they do actually the line that they're doing, then you can see that if you extrapolate the same thing to this 5% error rate, you do end up at something like 10 to the 18 flops. And they also compare this to the equivalent carbon dioxide emissions. For example, right now we are somewhere between the co2 generated by the average US resident in one year and the co2 generated by the average US resident in a lifetime, the current models somewhere in between to train them once if you actually extrapolate this to the 5% error rate to the 10 to the 18 flops, then it becomes suddenly co2 generated by New York City in one month. So the entire city of New York City for one month is the same as GPUs go brrrr to train ImageNet. Now that is pretty shocking, I have to say, you know, it checks out they have done the research, they extrapolated correctly here, and they come to this conclusion, the co2 equivalents, I'm sure they are measured correctly and so on. I do have several problems with this though. The first one I already said the zigzag in this graph right here doesn't really suggest that you can simply extrapolate over these advances. Also, the 2020 point seems to be quite out there. So if there was any architecture search involved, if there was any giant free training involved or anything like this, I'm sure like that that adds to the co2 emissions, but it doesn't say that you cannot achieve the same thing with something else. So whether the slope of the line is really the black one right here, or more like the blue one I drew, it makes quite a bit of a difference actually makes an exponential difference. So I'm a bit doubtful that you can really pinpoint this 5% error point to five years in advance. Okay, it's 2022 now, so three years, but still and speaking of co2 equivalents, not all energy is equal. For example, Google prides itself in being zero emission. Therefore, if Google trains a model, there is no co2 equivalent, presumably. Now I think carbon neutrality and zero emissions and words like this are sometimes a bit of a scam, but still not all energy is equal. And especially these large companies, they can distribute their workload across the planet to where the energy is used most efficiently. And lastly, and this I think should really the main point here is that we have made advances, none of these achievements here that we've made over the past years are only scaling up the scaling up always came with some sort of invention that made it more efficient or more viable to scale up residual networks, all of a sudden could scale to many, many more layers because of the invention of the residual connection or the addition depending on who you ask. So the residual networks became bigger and deeper without having to waste more computation. In fact, they had less parameters than many equivalent models of the time. So I don't think we should neglect the inventions we make along the way in order to scale up. Now, of course, people are always going to put in whatever flops they have in order to achieve the best possible number. But I think for most of these advances, it was really new inventions that triggers the usage of these flops rather than the other way around. And the authors of these articles actually agree a little bit, they say is it really reasonable to extrapolate like this and extrapolating this way would be unreasonable if we assume that researchers would follow this trajectory all the way to such an extreme outcome. We don't face with skyrocketing costs, researchers will either have to come up with more efficient ways to solve these problems, or they will abandon working on these problems and progress will languish, which is true. So rather than being a warning cry about we're going to waste an entire city's CO2 emissions for a month for one model, it's more of a warning against we're going to have to come up with new methods and different ways of training these models. And we can't rely on scale to bring us advances. They also give some money numbers right here. They said, for example, deep mind traded system to play go, it was about $35 million in cost. When they trained AlphaStar, they purposefully didn't try multiple ways of architecting an important component because the training cost would have been too high. In GPT three, they made a mistake, but they didn't fix it due to the cost of training, it wasn't feasible to retrain the model and so on. And also mentioning that GPT three cost about 4 million to train. Now, yes, of course, researchers that train these giant models comes with substantial costs. So you have to think twice if you really want to do your grid search and whatnot. So the experimentation methodology has become a bit different. But also, you have to keep in mind these big numbers $35 million, $4 million, and so on. First of all, this isn't really that much in comparison to what the people costs that worked on the model. And second of all, this is almost necessary. All of the models that we see today have cost substantially more in the past to train, but someone had to do it first, I can only train BERT today because Google has invested ginormous amounts of resources trying out how to train it training the first one at considerable cost. And only after that have other people jumped on prices have come down training got more efficient. And now I can do it from the comfort of my home essentially on a colab or on my home GPU. And isn't this the case with all inventions somehow at first, it's just a few, it's really expensive because it's custom because we haven't figured it all out yet. And then over time, cost will come down, efficiency will go up and the easiness is just much better. So rather than saying, Oh, wow, DeepMind spent $35 million. Oh, no, I'm like, cool, you know, since they're doing this two, three, four years, I will be able to do so for simply 2 million and pay, you know, so the article gives some solutions to that different avenues, though they are mostly a little bit pessimistic about most of them. So first of all, they said you can use specific processors designed specially for deep learning. Now the newest generations of GPUs are actually a little bit tuned to deep learning, but there are also tensor processing units. And there are a number of other hardware vendors that try to get into the space of specifically building chips for deep learning. What they criticize here is the fact that this hardware has to do trade offs, they have to increase specialization for generality. And also with specialization, you face diminishing returns. And of course, the more specialized you are, the less you can invent new things, because you're essentially locked into what the hardware can do. They also discuss training networks that are smaller, but they criticize that often this increases the training costs because you essentially train a big network and then you train again to make it smaller to distill it. And that's also not the solution to reducing training cost. But it might be a good solution if a model needs to be trained once and then largely runs in inference mode, such as GPT three, they also discuss meta learning where you essentially train a good initialization for a lot of problems. And then you transfer that initial solution to new problems. So if you have a good meta learner, they will be at an excellent starting point for solving new problems, therefore reducing the training cost in each of these new problems. But they also mentioned that and I agree meta learning is yet at the stage where it doesn't really work. The training you put into the initial meta learner doesn't often pay off to new problems. Yes, it works in papers. But in papers, you already know which other problems you're going to measure it on. So they say even small differences between the original data and where you want to use it can severely degrade performance. Now they also mentioned this paper right here, Benjamin Recht of the University of California, Berkeley and others have made this point even more starkly showing that even with novel data sets purposely constructed to mimic the original training data, performance drops by more than 10%. Now I want to highlight this a little bit because this talks about a paper called Do ImageNet classifiers generalize to ImageNet. This is also usually called ImageNet v2. Because what these authors did is they try to follow the protocol of the original ImageNet data collection as closely as possible and come up with a new test set, the so called ImageNet v2. It's not a training set, it's not a training set is just a test set. And they show pretty convincingly that for any classifier that performs in any way on ImageNet v1, its performance on ImageNet v2 will be something like 10 points lower, it's a fairly straight line. So this is what the article talks about. However, the article doesn't talk about this paper right here called identifying statistical bias in data set replication by MIT and UC Berkeley, which shows pretty convincingly that there is in fact, a difference between the data collection mechanism of ImageNet v1 and v2. It is a subtle difference, but there is a difference nonetheless, that difference makes it such that there is a significant difference in what kind of images are chosen for the two data sets. And when you correct for that difference, then this drop in accuracy for ImageNet v2 almost entirely vanishes. Now, okay, the article is right in first instance, there is a small difference between the original data and the new data, and that severely degrades performance. But this particular difference in performance is due to the new data set having a different methodology, and that directly makes the samples harder. It's not like the samples are different in some sort of a there are different kinds of images is that very directly because of how they collected them, they are more difficult to classify, it's the same data, but more difficult. So we shouldn't be surprised that performance drops by 10%. In this particular instance, I just thought it was interesting to mention since the article specifically focuses on this paper right here. And I don't think this paper is a good example of what they're trying to say. Okay, so what's the conclusion to all of this? Here is the final recommendation that the article makes to evade the computational limits of deep learning would be to move to other perhaps as yet undiscovered or underappreciated types of machine learning. And of course, what they mean is that they want to bring the insights of experts, which can be much more computationally efficient, and that we should maybe look at things like neuro symbolic methods and other techniques to combine the power of expert knowledge and reasoning with the flexibility often found in neural networks. Now, why does every discussion about the scaling of deep learning always end with Well, we should use more expert systems and reasoning and logic and the neural networks don't understand anything. Now granted, it is okay to suggest this, it's probably a good way forward. But as of yet, as of now, the neuro symbolic systems are actually just the expert systems as well. They are so so not good. And of course, that's the case with any young research topic. But just because something is computationally efficient, it doesn't mean that we should switch to that because of it. Now I'd be super duper happy if symbolicism makes a comeback if we could somehow combine algorithms and deep learning, if we could combine reasoning and knowledge bases and input from domain experts and all of this. But as of today, that is not really a benefit, it's more like a substitute. So you can make machine learning more efficient by inputting lots and lots of priors from domain experts. That's completely cool. But what we've seen over and over and over again is that as soon as you give the ML system enough data, it starts to outperform these experts. And I think what I'd like to see from a neuro symbolic system or anything like this is that in fact, it does outperform even the most data hungry machine learning methods that the symbolicism is not just a substitute for more data, but an actual improvement over any data that I could find. And that's just something that I personally haven't seen, you might disagree, but I haven't seen a convincing argument yet that that is the case for any of the symbolic systems we have today. computational efficiency alone is simply not enough. But hey, tell me what you think. What do you think about this article? Do you agree with them? Do you not agree with them? I'll link the full article in the description, give it a read if you want and subscribe. I'll see you next time. Bye bye.
 Hi there, I saw this article in IEEE spectrum called deep learnings diminishing returns,
 the cost of improvement is becoming unsustainable. This is by Nielse Thompson, Christian Greenwald,
 Kihon Lee, and Gabrielle F. Monso. And I thought it was an interesting read, because it talks about
 the computational limits that we're reaching with deep learning today. And I have it over here in
 an annotatable form, though it might not look as pretty. I think the article, it leads up to the
 point where it shows just how much compute will be needed to make further improvements in deep
 learning and what the consequences of that might be, and some of the ways that people are trying to
 get around it. Now, I don't agree with everything the article says, but I think it's a it's a pretty
 neat read. It's pretty short. So I thought we can talk about it a little bit. So the article starts
 out with essentially praising deep learning for achieving so many things, for example, translating
 between languages, predicting how proteins fold, and many other things playing games as complex as
 go. They say it has risen relatively recently, but it has a long history. They mentioned 1958.
 And Frank Rosenblatt at Cornell, they designed the first artificial neural network, they say
 Rosenblatt's ambitions outpaced the capability of his era. And he knew it. Apparently, he said,
 as the number of connections in the network increases, the burden of a conventional digital
 computer soon becomes excessive. So why are deep neural networks working? Because of course,
 computers have increased in power massively, just for computing power, there has been whatever a 10
 million fold increase, according to Moore's law. And that's usually just measured in something like
 CPU instructions. And now we went even beyond that building special purpose hardware such as GPUs,
 which aren't actually special purpose for this, but also TPUs. So they say these more powerful
 computers have made it possible to construct networks with vastly more connections and neurons
 enhance greater ability to model complex phenomena. And of course, these are the deep neural networks
 that power most of today's advances in AI, they draw a comparison right here, they say like
 Rosenblatt before them, today's deep learning researchers are nearing the frontier of what
 their tools can achieve, essentially claiming that we are in a similar situation today, we have the
 models that can achieve things, and we know pretty much that scaling them up can increase
 performance. However, we're kind of at the limits of how much we can scale. For example, I reported
 on this that Sam Altman apparently said GPT four will not be much bigger than GPT three, it will
 be trained more efficiently, we'll have some smartness in it on how it's processed, it will
 use more compute, but it will not necessarily be that much bigger in scale. So the first thing the
 article touches about deep learning is the fact that deep networks are over parameterized. For
 example, the noisy student model has some 480 million parameters, yet is trained on only 1.2
 million labeled images, which is the image net data set. Now, of course, the noisy student model,
 if I understand correctly, also may leverage unlabeled data. But granted, today's neural
 networks are massively over parameterized, they have more parameters than data points available,
 therefore, they should horribly overfit. But they don't. They say classically, this would lead to
 overfitting where the model not only learns general trends, but also the random vagaries of the data
 it was trained on. Deep learning avoids this trap by initializing the parameters randomly,
 and then iteratively adjusting sets of them to better fit the data using a method called
 stochastic gradient descent. Surprisingly, this procedure has been proven to ensure that the
 learned model generalizes well. Now, I'm pretty sure that we are not yet sure why exactly deep
 networks don't overfit or why they generalize as they get over parameterized. I know there are some
 proofs around SGD and so on. But these proofs usually require assumptions that just make them
 completely lose touch to reality. But the core message is true, deep networks are over parameterized.
 And that is probably one of the reasons why they work so well. And being over parameterized, they
 are quite flexible. They say at the good news is that deep learning provides enormous flexibility.
 The bad news is that this flexibility comes at an enormous computational cost. This unfortunate
 reality has two parts. They say the first part is true of all statistical models to improve
 performance by factor of k, at least k squared more data points must be used to train the model.
 Does this really hold for all statistical models? Is this from the same theory that says the
 statistical models should overfit when they're over parameterized? I'm not sure. The second part,
 they say, of the computational cost comes explicitly from over parameterization. Once accounted for,
 this yields a total computational cost for improvement of at least k to the fourth power,
 meaning for a tenfold improvement, you would need to increase the computation by 10,000. Now,
 regardless of whether you think the theoretical analysis is actually accurate here, again,
 this is from the same area that says these models should overfit horribly, it doesn't matter,
 because these people have actually collected data. And they say theory tells us that computing needs
 to scale with at least the fourth power of the improvement in performance. In practice,
 the actual requirements have scaled with at least the ninth power. So when you actually measure how
 much people need to scale computation in order to achieve a given performance, then it's actually
 it's much worse than the theory predicts. In fact, they have these neat graphs right here. So on the
 left, you can see the percent error, I believe this is the image net classification data set.
 And on this axis, you can see the time. Now here you can see that over time, as time progresses,
 the error has come down and down and down again, as new state of the art models were proposed
 ever since the 2012 success of Alex net. And if you extrapolate that you can pretty clearly see
 that around 2025, we should be at approximately 5% of error. See, I thought you'd had to actually
 do something to reach a new state of the art on image net. But as it turns out, we just need to
 sit here and wait until 2025. Okay, jokes aside, they overlay this graph with another graph right
 here. And that is the comparison of again, percent error on the y axis. But now it's not the year in
 which the achievement was made. But it is number of computations in billions of flops. And notice the
 log scale down here. Now I have to say this graph right here makes it pretty clear that there might
 be something like a relationship, even maybe a linear relationship that you can extrapolate
 right here. I'm not so sure like these models are up here and then goes like here and then it goes
 here and then it goes here. And then it goes over here to 2020. And really without that, you
 probably have a line that goes something like this. Now in any case, if they do actually the
 line that they're doing, then you can see that if you extrapolate the same thing to this 5%
 error rate, you do end up at something like 10 to the 18 flops. And they also compare this to
 the equivalent carbon dioxide emissions. For example, right now we are somewhere between
 the co2 generated by the average US resident in one year and the co2 generated by the average
 US resident in a lifetime, the current models somewhere in between to train them once if you
 actually extrapolate this to the 5% error rate to the 10 to the 18 flops, then it becomes suddenly
 co2 generated by New York City in one month. So the entire city of New York City for one month
 is the same as GPUs go brrrr to train ImageNet. Now that is pretty shocking, I have to say,
 you know, it checks out they have done the research, they extrapolated correctly here,
 and they come to this conclusion, the co2 equivalents, I'm sure they are measured correctly
 and so on. I do have several problems with this though. The first one I already said the zigzag
 in this graph right here doesn't really suggest that you can simply extrapolate over these
 advances. Also, the 2020 point seems to be quite out there. So if there was any architecture search
 involved, if there was any giant free training involved or anything like this, I'm sure like that
 that adds to the co2 emissions, but it doesn't say that you cannot achieve the same thing with
 something else. So whether the slope of the line is really the black one right here, or more like
 the blue one I drew, it makes quite a bit of a difference actually makes an exponential difference.
 So I'm a bit doubtful that you can really pinpoint this 5% error point to five years in advance.
 Okay, it's 2022 now, so three years, but still and speaking of co2 equivalents, not all energy is
 equal. For example, Google prides itself in being zero emission. Therefore, if Google trains a model,
 there is no co2 equivalent, presumably. Now I think carbon neutrality and zero emissions and
 words like this are sometimes a bit of a scam, but still not all energy is equal. And especially
 these large companies, they can distribute their workload across the planet to where the energy is
 used most efficiently. And lastly, and this I think should really the main point here is that
 we have made advances, none of these achievements here that we've made over the past years are
 only scaling up the scaling up always came with some sort of invention that made it more efficient
 or more viable to scale up residual networks, all of a sudden could scale to many, many more layers
 because of the invention of the residual connection or the addition depending on who you ask. So the
 residual networks became bigger and deeper without having to waste more computation. In fact, they
 had less parameters than many equivalent models of the time. So I don't think we should neglect
 the inventions we make along the way in order to scale up. Now, of course, people are always
 going to put in whatever flops they have in order to achieve the best possible number. But I think
 for most of these advances, it was really new inventions that triggers the usage of these flops
 rather than the other way around. And the authors of these articles actually agree a little bit,
 they say is it really reasonable to extrapolate like this and extrapolating this way would be
 unreasonable if we assume that researchers would follow this trajectory all the way to such an
 extreme outcome. We don't face with skyrocketing costs, researchers will either have to come up
 with more efficient ways to solve these problems, or they will abandon working on these problems
 and progress will languish, which is true. So rather than being a warning cry about we're
 going to waste an entire city's CO2 emissions for a month for one model, it's more of a warning
 against we're going to have to come up with new methods and different ways of training these
 models. And we can't rely on scale to bring us advances. They also give some money numbers
 right here. They said, for example, deep mind traded system to play go, it was about $35 million
 in cost. When they trained AlphaStar, they purposefully didn't try multiple ways of
 architecting an important component because the training cost would have been too high.
 In GPT three, they made a mistake, but they didn't fix it due to the cost of training,
 it wasn't feasible to retrain the model and so on. And also mentioning that GPT three cost about
 4 million to train. Now, yes, of course, researchers that train these giant models comes
 with substantial costs. So you have to think twice if you really want to do your grid search
 and whatnot. So the experimentation methodology has become a bit different. But also, you have
 to keep in mind these big numbers $35 million, $4 million, and so on. First of all, this isn't
 really that much in comparison to what the people costs that worked on the model. And second of all,
 this is almost necessary. All of the models that we see today have cost substantially more in the
 past to train, but someone had to do it first, I can only train BERT today because Google has
 invested ginormous amounts of resources trying out how to train it training the first one at
 considerable cost. And only after that have other people jumped on prices have come down training
 got more efficient. And now I can do it from the comfort of my home essentially on a colab or on
 my home GPU. And isn't this the case with all inventions somehow at first, it's just a few,
 it's really expensive because it's custom because we haven't figured it all out yet. And then over
 time, cost will come down, efficiency will go up and the easiness is just much better. So rather
 than saying, Oh, wow, DeepMind spent $35 million. Oh, no, I'm like, cool, you know, since they're
 doing this two, three, four years, I will be able to do so for simply 2 million and pay, you know,
 so the article gives some solutions to that different avenues, though they are mostly a
 little bit pessimistic about most of them. So first of all, they said you can use specific
 processors designed specially for deep learning. Now the newest generations of GPUs are actually
 a little bit tuned to deep learning, but there are also tensor processing units. And there are a
 number of other hardware vendors that try to get into the space of specifically building chips for
 deep learning. What they criticize here is the fact that this hardware has to do trade offs,
 they have to increase specialization for generality. And also with specialization,
 you face diminishing returns. And of course, the more specialized you are, the less you can
 invent new things, because you're essentially locked into what the hardware can do. They also
 discuss training networks that are smaller, but they criticize that often this increases the
 training costs because you essentially train a big network and then you train again to make it smaller
 to distill it. And that's also not the solution to reducing training cost. But it might be a good
 solution if a model needs to be trained once and then largely runs in inference mode, such as GPT
 three, they also discuss meta learning where you essentially train a good initialization for a lot
 of problems. And then you transfer that initial solution to new problems. So if you have a good
 meta learner, they will be at an excellent starting point for solving new problems, therefore reducing
 the training cost in each of these new problems. But they also mentioned that and I agree meta
 learning is yet at the stage where it doesn't really work. The training you put into the initial
 meta learner doesn't often pay off to new problems. Yes, it works in papers. But in papers,
 you already know which other problems you're going to measure it on. So they say even small
 differences between the original data and where you want to use it can severely degrade performance.
 Now they also mentioned this paper right here, Benjamin Recht of the University of California,
 Berkeley and others have made this point even more starkly showing that even with novel data
 sets purposely constructed to mimic the original training data, performance drops by more than 10%.
 Now I want to highlight this a little bit because this talks about a paper called Do ImageNet
 classifiers generalize to ImageNet. This is also usually called ImageNet v2. Because what these
 authors did is they try to follow the protocol of the original ImageNet data collection as closely
 as possible and come up with a new test set, the so called ImageNet v2. It's not a training set,
 it's not a training set is just a test set. And they show pretty convincingly that for any classifier
 that performs in any way on ImageNet v1, its performance on ImageNet v2 will be something
 like 10 points lower, it's a fairly straight line. So this is what the article talks about.
 However, the article doesn't talk about this paper right here called identifying statistical bias in
 data set replication by MIT and UC Berkeley, which shows pretty convincingly that there is in fact,
 a difference between the data collection mechanism of ImageNet v1 and v2. It is a subtle
 difference, but there is a difference nonetheless, that difference makes it such that there is a
 significant difference in what kind of images are chosen for the two data sets. And when you correct
 for that difference, then this drop in accuracy for ImageNet v2 almost entirely vanishes. Now,
 okay, the article is right in first instance, there is a small difference between the original data
 and the new data, and that severely degrades performance. But this particular difference
 in performance is due to the new data set having a different methodology, and that directly makes
 the samples harder. It's not like the samples are different in some sort of a there are different
 kinds of images is that very directly because of how they collected them, they are more difficult
 to classify, it's the same data, but more difficult. So we shouldn't be surprised that performance
 drops by 10%. In this particular instance, I just thought it was interesting to mention since the
 article specifically focuses on this paper right here. And I don't think this paper is a good
 example of what they're trying to say. Okay, so what's the conclusion to all of this? Here is the
 final recommendation that the article makes to evade the computational limits of deep learning
 would be to move to other perhaps as yet undiscovered or underappreciated types of machine
 learning. And of course, what they mean is that they want to bring the insights of experts,
 which can be much more computationally efficient, and that we should maybe look at things like
 neuro symbolic methods and other techniques to combine the power of expert knowledge and reasoning
 with the flexibility often found in neural networks. Now, why does every discussion about
 the scaling of deep learning always end with Well, we should use more expert systems and reasoning
 and logic and the neural networks don't understand anything. Now granted, it is okay to suggest this,
 it's probably a good way forward. But as of yet, as of now, the neuro symbolic systems are actually
 just the expert systems as well. They are so so not good. And of course, that's the case with any
 young research topic. But just because something is computationally efficient, it doesn't mean
 that we should switch to that because of it. Now I'd be super duper happy if symbolicism makes a
 comeback if we could somehow combine algorithms and deep learning, if we could combine reasoning
 and knowledge bases and input from domain experts and all of this. But as of today,
 that is not really a benefit, it's more like a substitute. So you can make machine learning
 more efficient by inputting lots and lots of priors from domain experts. That's completely
 cool. But what we've seen over and over and over again is that as soon as you give the ML system
 enough data, it starts to outperform these experts. And I think what I'd like to see from a
 neuro symbolic system or anything like this is that in fact, it does outperform even the most
 data hungry machine learning methods that the symbolicism is not just a substitute for more
 data, but an actual improvement over any data that I could find. And that's just something that I
 personally haven't seen, you might disagree, but I haven't seen a convincing argument yet
 that that is the case for any of the symbolic systems we have today. computational efficiency
 alone is simply not enough. But hey, tell me what you think. What do you think about this article?
 Do you agree with them? Do you not agree with them? I'll link the full article in the description,
 give it a read if you want and subscribe. I'll see you next time. Bye bye.