The Hardware Lottery (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper hardware gpus tpus gpu tpu convolutional neural networks yann lecun history historic ai winter expert systems babbage google accelerators cuda nvidia flops von neumann architecture bottleneck parallelize research funding society cost competition general purpose fpga
#ai #research #hardware

We like to think that ideas in research succeed because of their merit, but this story is likely incomplete. The term "hardware lottery" describes the fact that certain algorithmic ideas are successful because they happen to be suited well to the prevalent hardware, whereas other ideas, which would be equally viable, are left behind because no accelerators for them exists. This paper is part history, part opinion and gives lots of inputs to think about.

OUTLINE:
0:00 - Intro & Overview
1:15 - The Hardware Lottery
8:30 - Sections Overview
11:30 - Why ML researchers are disconnected from hardware
16:50 - Historic Examples of Hardware Lotteries
29:05 - Are we in a Hardware Lottery right now?
39:55 - GPT-3 as an Example
43:40 - Comparing Scaling Neural Networks to Human Brains
46:00 - The Way Forward
49:25 - Conclusion & Comments

Paper: https://arxiv.org/abs/2009.06489
Website: https://hardwarelottery.github.io/

Abstract:
Hardware, systems and algorithms research communities have historically had different incentive structures and fluctuating motivation to engage with each other explicitly. This historical treatment is odd given that hardware and software have frequently determined which research ideas succeed (and fail). This essay introduces the term hardware lottery to describe when a research idea wins because it is suited to the available software and hardware and not because the idea is superior to alternative research directions. Examples from early computer science history illustrate how hardware lotteries can delay research progress by casting successful ideas as failures. These lessons are particularly salient given the advent of domain specialized hardware which makes it increasingly costly to stray off of the beaten path of research ideas.

Authors: Sara Hooker

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi there. Are you interested in winning the lottery? Then let me tell you this video is not for you. This video is not about winning the lottery. Okay? I've done enough videos with lottery in the title only for people to be mad at me for not telling them how to win the lottery. This is about computer science research and very unfortunately the author of this paper has decided to put this word in the title. So if you're here because you want to win the lottery, this is not for you. It's something completely different. For everyone else today we're looking at the hardware lottery by Sarah Hooker of Google Brain. This paper is it's kind of a mix. It's part of a historic look back at hardware and software developments in machine learning and it is a analysis of kind of the current situation and an outlook and sort of an opinion piece of the way forward and how hardware and software should mix and what we should focus on in the future. So the basic the basic principle is quite simple in this paper. It introduces this term the hardware lottery. This essay introduces the term hardware lottery to describe when a research idea wins because it is compatible with available software and hardware and not because the idea is superior to alternative research directions. So right off the bat I think this is a statement where I think many people can agree or I think almost everyone will some agree with this statement in to to a certain degree but certainly to a high degree right we are all aware that of course we have the hardware we have hardware is very inflexible it's expensive to develop and so on so any sort of software development any algorithmic development may simply succeed because it is suited to the hardware that we have. So that was my first reaction when I read this paper it's a it's a it's a very gut feeling of yes of course this is the case but then the historic analysis is also nice but I was wondering what is there a deeper reason to to kind of go into this and we are going to see some pros and cons that I think in this paper right here where it I'm not exactly entirely sure what specific point is trying to make the overarching point I completely agree with the fact that of course what hardware is here is important and may lead to certain ideas succeeding but it I have I have a trouble with the narrower points and I'm gonna try to illustrate this in this paper while also telling you what the paper says. So first of all here the term is called the hardware lottery but off the bat you already see that it says a research idea wins because it is compatible with available software and hardware so the hardware lottery right off the bat is connect is means that also the software is there so it's technically the hard and software lottery and the bigger the bigger question I would have to someone arguing that really the hardware lottery is an important concept to have is why what does what distinguishes the hardware lottery let's let's even say it's just hardware what distinguishes the hardware lottery from any lottery like why can't I say okay there's the X lottery and the X lottery is is any circumstance any circumstance is that that surrounds a research idea right here we have idea one idea two idea three and they all depend on many circumstances and X is one of those circumstances and it just so happens that the circumstance in the world favors idea two and a different circumstance would actually favor idea one what's so special about hardware other than it's more expensive than software right to to to illustrate this further let's say okay you have you have hardware and you say well hardware is expensive but then again you can sort of build a hierarchy where okay down here there is like ideas they depend on software like software frameworks that we have such as TensorFlow pytorch these again depend on particular hardware but and you can say okay the hardware is much more expensive so we are not as flexible and the ideas might just succeed because of the hardware but then you can go even step further and say well up here is sort of the consumer if you don't like the market term then maybe say the society the end user and so on because the hardware ultimately is directed towards what humans in society need and that changes over time as well so and and it's it's way more expensive to change the needs of human society than to change the hardware so I can just also claim okay X is now society so the one particular research idea down here might win simply because it is more suited to the current societal needs and that kind of carries over and you might say well make doesn't that make it a good idea doesn't that make it preferable to idea idea to preferable to idea three over here that would just optimize for a different society which leads us to the question what does it mean to first what does it mean to win here it just says a research idea wins and you might have an idea so I've I have an idea it's not clearly defined here but maybe winning means that a lot of researchers actually research in that direction and the other question is here and not because the idea is superior to alternative research directions and here my question would be what does superior mean what does it what does it mean for an idea to be superior as I said here certainly if an idea is more incongruent with current societal needs you might claim it's superior and someone else might say well if societal needs were different than a different research idea might be suited better the same way someone could say well if hardware was different than a different research idea might be better maybe you can say if hardware was different a different research idea might be better suited to the current needs of society but then I'm pretty sure I can go two three four levels up here again so these these terms are a bit vague I think we can all the again the initial the initial sentiment when reading this is absolutely in favor right I absolutely agree I don't want to want to trash this I just want to sort of I try to think a bit deeper about what is actually said here and this is where sort of my my troubles start so let's dig a bit into the historic part and I think the point the paper is sort of trying to make is that not yet that there are specific hardware choices that were made at one particular point and because it's so expensive to change hardware that means that a lot of researchers simply go along with whatever ideas work on that particular hardware that's available and other research ideas are neglected simply because the hardware isn't available which again this is a sentiment that I think we can we can all agree with so the first part here the paper is in the following sections and this is a been important to keep in mind as a red thread because I feel one can get lost in the details of the paper so in the first section section two we ask what has incentivized the development of software hardware and machine learning research in isolation we need to read this first this essay begins by acknowledging a crucial paradox machine learning researchers mostly ignore hardware despite the role it plays in determining what ideas succeed so the argument is that we we develop ideas independent of hardware but also we don't it kind of makes it a double double point it says that we think we just think about ideas but the ideas we might think about may be shaped by the hardware that's available and if we're not aware of that we might not we might not see other ideas as viable so section two asks what has incentivized the development of software hardware and machine learning research in isolation so where does this come from that we don't think about the hardware that's at the end section three considers the ramifications of this siloed evaluation with examples of early hardware and software lotteries so this is the kind of risk historical look back then today the hardware landscape is increasingly heterogeneous this essay posits that the hardware lottery has not gone away and the gap between the winners and the losers will grow increasingly larger so this is a point that the paper basically makes that this hardware lottery has not gone away so right now we are in this hardware lottery and it does so specifically with regards to saying that chips like GPUs and TPUs and even more specialized chips are optimized to neural networks and that's why the whole world sort of over focuses on neural networks right now and discards other research ideas and the gap between the winners and the losers will grow increasingly larger meaning that the research ideas that are seen as in viable now if we develop even more hardware into that direct into the direction of neural networks those research ideas will become more and more inaccessible to the community then lastly sections four to five unpack these arguments so the ones that we've just seen section six concludes with some thoughts on what it will take to avoid future hardware lotteries all right so section two here is this sort of historic look back and it goes from these it the point is here separate tribes so the point is that something has made it such that the communities the software communities and the hardware communities and the idea let's say the idea communities the researchers in AI algorithms let's call them the algorithmers they they they don't think that much about each other and it makes the case that early machines were super duper specialized early machines were single use were not expected to be repurposed for a new task because of the cost of electronics and the lack of cross-purpose software so early machines early computing machines were just single purpose and so on but that all changed when the whole world focused on sort of general purpose CPUs that could execute any instructions of course according to Turing machine or von Neumann architectures so the point that the paper makes is at some point a shift happened the general purpose computer area crystallized in 1969 when an opinion piece by young engineer called Gordon Moore appeared in electronics magazine with the app title cramming more components onto circuit boards that's a cool title so this famously gave rise to Moore's law or predicted you could double the amount of transistors on an integrated circuit every two years and this sort of held true where people stopped building general like sorry people stopped building special-purpose hardware but invested just more and more and more into building these general-purpose chips these CPUs that and the reason why they stopped making specialized hardware is any specialized hardware you build will simply be surpassed by the next generation of CPUs so even if you make a specific purpose hardware for some problem you just have to wait like one or two of these cycles and ordinary general-purpose CPUs will simply have will overtake your specialized hardware and since CPUs are general purpose the market for them is naturally huge so this this has made it such that what was mainly developed was general-purpose CPUs I think the paper wants to make the point though I'm not in exactly sure I think it wants to make the point that even though the CPUs might be called general-purpose they aren't general-purpose like they have their specific advantages and disadvantages and that's going to hurt for example neural networks in the years following this so in conclusion to this chapter they say in the absence of any lever with which to influence hardware development machine learning researchers rationally began to treat hardware as a sunk cost to work around rather than something fluid that could be shaped however just because we have abstracted away hardware does not mean it has ceased to exist early computer science history tells us there are many hardware lotteries where the choice of hardware and software has determined which idea succeeded and which fail and the example is kind of the Charles Babbage's analytic engine that Charles Babbage designed but was something like 50 years earlier or so then parts could even be manufactured for this idea to succeed and we know many stories of these people being ahead of their time and they have this interesting quote I think somewhere from Silicon Valley here being too early is the same as being wrong and this paper of course focuses on hardware but to come back the conclusion of this chapter is that because of this general purpose area because the entire focus was on building general purpose CPUs this has led to people not really having integrated thought of hardware software algorithm but treating hardware as this thing that can execute any instruction and then the the algorithm comes on top of this sort of black box that we can't really change we just have the hardware we have yeah which which comes back I'm and again I'm not sure like sure that that sure I agree that the entire world focusing on general purpose CPUs has some influence but certainly hardware is just expensive to make so you could argue that even if this hadn't happened a machine learning researcher wouldn't necessarily think about the hardware but they would at least have a choice if there were a selection of hardwares right okay so that was the section 2 section 3 now we really go into the historic evidences and there are kind of early historic evidence like this Charles Babbage's machine that he invented an early example the analytical machine in 1837 and and no it wasn't even decades it was only surface during World War two in the first part of the 20th century electronic vacuum tubes were heavily used were heavily used for heavily used this is I've not I've noticed a number of typos in in the paper I realized it's pre-print if the author is listening I can also I can also make a list but this this one just popped out for radio communication and radar during World War two these vacuum tubes repurposed to provide the compute power necessary to break the German enigma code so it would be long after not only after Charles Babbage invented this machine but even after he died that people would would sort of re-take and in some parts reinvent his ideas to to build modern computers the big example though that the paper makes is what it calls the lost decades and this is the story of neural networks coupled with two things with an AI winter and a focus on expert systems and maybe also though that's not entirely mentioned here a focus on things like SVMs so I think it's widely known that the main ingredients for neural networks are very very very old so here the paper gives some examples back propagation invented in 63 reinvented reinvented again and deep convolutional networks paired with back propagation by on the car it says however it was only three decades later that deep neural networks were widely accepted as a promising research direction I think this this sort of the timeline here is this here probably refers to around 2010 shortly after that of course Alex net beats image net and so on but even earlier a bit earlier people were doing heavy research into neural networks and three decades later so this is paired with kind of these numbers right here let's say 1970 1980 when these ideas were invented presented but computers back then were simply unsuited to the to run neural networks here it says the gap between these algorithmic advances and empirical successes in large part to to incompatible hardware during the general purpose computing areas hardware like CPUs were heavily favored and widely available CPUs were good at executing any set of complex instructions but occur high memory costs because of the need to cache intermediate results and process one instruction at a time this is known as the von neumann bottleneck the available compute is restricted by the lone channel between CPU and memory along which data has to travel sequentially so the paper goes on and says there were some efforts into specialized hardware for neural networks but funding was kind of not there and other specialized hardware was more into the direction of popular ideas then like prologue and lisp which could do expert systems and not necessarily neural networks and only only it would take a hardware fluke in the early 2000s a full four decades after the first paper about back propagation was published for the insight about massive parallelism to be operationalized in a useful way for connectionist deep neural networks a graphical processing unit was originally introduced in the 1970s as a specialized accelerator for video games and developing graphics yada yada yada GPUs were repurposed for an entirely unimagined use case to train deep neural networks had one critical advantage over CPUs they were far better at parallelizing a set of simple decomposable instructions such as matrix multiplications multiples multiplications multiples I don't know so the the point here is that the ideas were around for a long time but it would take GPUs to make them work and so the the image that the paper builds up I think is that you have these you're you're here and you research and then you have a decision to make which hardware do I build for the future and there are two directions this is direction one and this is direction two and let's say for whatever reason direction one is chosen okay then because it's so expensive to build different hardware the the world largely goes with direction one and builds on top of that okay so that also means that all the research ideas that profit from direction one will appear to be much more effective that research ideas that would have profited from direction two and it sort of says that neural networks are over here and it's sort of the and the the let's say the other systems what do we give expert systems let's call them expert systems and other types of ideas were over here and they appear to work really well until they stopped in progress and then by accident sort of this road here was traveled use with GPU so it was not obvious but by accident still this was developed and then neural networks could flourish and if it wasn't for that fluke if it wasn't for video games basically or animation we would have never known that neural networks work as well as they do so again that's the point the paper makes and I think we can all agree with that particular point but I want to again I want to build up sort of a different picture right here in that why why is only like I feel hardware is considered a bit much here so I think you can make the general case that at any junction you have several things you can choose and then once you choose a thing all the things go in that direction like new ideas will be more in that direction also new hardware will be more in that direction because a lot of people research on it the paper also makes the point there's kind of this feedback loop but let's say neural networks were down here what I what I would argue and this is a bit of a point the paper makes in in a half half formulated way I think is that it basically says that had we had we invested in matrix multipliers in GPUs instead of CPUs in these early years that means that neural networks would have sort of succeeded as an idea at that time and I'm not entirely convinced of this because first of all you can see right here GPUs were actually around in the 1970s so the hardware was was available it's not it's not like it was super easy in in 2010 it was for these early researchers to build their code into GPU compatible code that was certainly hard especially if you read the papers but it would have been hard in 1970 as well it would not have been significantly harder I think so I I'm not sure if the picture is really like this or if the picture so if this is the CPU direction is more like that neural networks are actually somewhere up here and the fact is we we we actually needed the good CPUs in order to develop day in order to make use of the GPUs right and this here would be GPU in order to make use of the GPUs to then enable these neural networks on the GPUs because certainly it has it has helped a lot that CPUs were built that you know computers just built on GPUs would be sad computers computers built on CPUs are cool they can do multi-processing they can do internet they can do actually they can do most of the video game except display the graphics and very arguably that without the heavy focus on CPUs we would not have neural networks today even if we had invested all of that effort into building GPUs because society has just advanced so much because of CPUs so I'm sort of tempted to challenge this notion here that just because of the the happenstance that CPUs were advanced at that time that neural networks are they didn't have their breakthrough back then I think we needed both that being said I do agree with the paper that we might have never ever realized that neural networks worked if it weren't for the fact that there is specialized hardware around yeah so so that would be my my points to this the paper makes yeah makes this point about okay there is hardware lotteries and in so now it also introduces software lotteries though it said at the beginning that hardware lotteries included software but I'm going to guess that the general concept of a lottery was simply presented and again I I don't see exactly what's so special about hardware because again I can make the same case for software it's just a shorter time frame I can make the same case for theory right like whatever now neural tangent kernels are are are the hit right everyone's like wow NTKs blah blah blah blah blah who knows right but some big names announced this and some theory has been done in this direction and because there is already a big momentum lots of people publish in it who who knows if that's if that's a good idea if there were other ideas that had we done the fundamental work in this would flourish right now they I again I don't I agree with the sentiment I don't see why the hardware is the why the hardware is is such a special case right here so the next thing that the paper looks like it is kind of the current day so it tries to make the point that we might be in a hardware lottery right now and again the the intuition of course is yes of course we have the hardware we have it's difficult to change especially since hardware builds upon hardware with the tree I drew before let's draw it again you draw a tree and literally every decision you make in the tree and this doesn't only need to be hardware right every single decision you make will mean that pretty much all of the previous choices here are now fixed and ingrained we build upon we build upon inventions of the past it's impossible to go back and do all of these things again and if you see something curious right here and this is where we're going to later I want you to see what happens if here here is a good idea like here is my super duper booper idea and my super duper booper idea simply didn't make the cut for that choice like someone chose a different hardware direction software direction software library direction whatnot it wasn't in vogue and my idea was unpopular then if one choice is made this choice right here it's it's hard to go back if two choices are made right that build upon each other it's even harder to go back so as time goes on it's harder and harder and harder to go back which is a point that the paper will make at the end that the difference between the winners and the losers is getting bigger and bigger which is an effect that this idea that once was a curiosity that could be investigated becomes a very costly investigation because we need to reinvent and re-engineer a whole bunch of decisions and it at with time goes on it's simply forgotten because there's so much that we have built past this however this is for the loser right this is the loser however for the winner I I disagree right here because here it says okay this direction the idea direction here let's say there is a super cool idea that would beat neural the crap out of neural networks what not whatever whatever the latest schmidhuber paper is that that idea would beat neural networks and this here is neural networks and everyone's doing neural networks and schmidhuber idea is just forgotten about now to say that neural networks are the winner and the winners will increase and increase and increase is correct but it forgets that right here there is this whole branching so within the neural networks you have again this branching and maybe over here what kind of neural networks were completely forgotten like MLPs no MLPs are maybe still a thing I don't even remember like early early neural networks were 10 H nonlinearities for MLPs or something like this 9 by 9 filters 9 by 9 filters in convolution things like this right we it's sort of the 9 by 9 filters are technically in the class of neural networks but as time progresses and this branch here are the 3 by 3 filters which are massively out competing the 9 by 9 filters so the 9 by 9 filters are forgotten and it could be that if the 9 by 9 filters no sorry because of the 3 by 3 filters now we have specialized hardware that is exclusively focuses on 3 by 3 filters so we go down this route down this route down this route down this route and there might have been some other super duper idea down here that only works when we have really big filters and now we never know that this existed right so they say that the difference between the winners and the losers gets bigger and bigger sort of misjudges that these winners will be fractionated and fractionated and fractionated and every push in one direction comes with costs to these other directions within that winner branch but this is I don't yeah ultimately you know you have a choice you have a choice do I want to go back and go this direction or do I want to add something here it might just might be worth more for society to go up here the paper is going to argue at the end that we should sort of keep funding alternative directions in hardware which I think is always a good thing to not lock in on particular ideas but also you can you sort of have a have to strike a balance because you know researching on things that already work and make them better is a crucial part as well because you can discard these sub ideas that don't make any sense all right so it gives some examples of current hardware lottery winners to improve efficiency there is a shift from task agnostic hardware like CPUs to domain specialized hardware that tailor the design to make certain tasks more efficient the first examples of domain specific hardware at least over the last few years TPUs and then it also says edge TPUs Cortec arm Cortex m55 Facebook's Big Sur which I think is just like a box with a GPUs in it and some Infini band optimize explicitly for costly operations common to deep neural networks like matrix multiplies so here I have again there's there's this double meaning so it says here is task agnostic hardware like CPUs but at the same time it argues that CPUs are particularly bad at matrix matrix multiplies it's not really task agnostic it's just focused on on different tasks but I see what the what the paper means right here we do build hardware that make matrix multiplies faster which means that neural networks that benefits neural networks research closer collaboration between hardware and research communities will undoubtedly continue to make the training and deployment of deep neural networks more efficient for example unstructured pruning and weight quantization a very successful compression techniques in deep neural networks but in are incompatible with current hardware and compilations and compilations kernels hardware and compilations kernels I don't know what that means but it's incompatible with current hardware the paper argues that because we see that these ideas are good there will be specialized hardware for them and I think the point the papers trying to make is sort of like see another win for neural networks because we go down the neural network road people focus on neural networks focus on how to prune them and so on hardware will be developed which will lock us in further into neural networks which again is papers basically saying like look because we went this road right here we're gonna go this road a lot more but then what you have to see is that if we in if we then from this road go here because we do want to do weight quantization in this particular way we also are going to neglect this which would be doing some whatever other thing that we could do yeah so there's always there's always in each decision there's a branching undoubtedly the paper is correct and it says the branching decides the future but I think the focus here on hardware and neural networks versus non neural networks is a bit it's very specific to that thing it then it makes the it makes the point why it matters so why it matters it matters because the paper says okay where is that here in 2019 the paper was published called machine learning is stuck in a rut the authors consider the difficulty of training a new type of computer vision architecture called capsule networks and I kind of realized that capsule networks aren't really suited to current to current to current hardware and he says whether or not you agree that capsule networks are the future of computer vision the authors say something interesting about the difficulty of trying to train a new type of image classification architecture on domain specific specialized hardware hardware design has prioritized delivering on commercial use cases while built-in flexibility to accommodate the next generation of research ideas remains a distant secondary consideration which is true though I would also say I mean GPU CPUs and GPUs combined are extremely general operations like they're very very generalized okay GPUs are good at matrix multiplies but CPUs are good at a lot of other things I would say the GPU CPU combo is a very very very flexible general-purpose hardware design that doesn't doesn't lock you in too much and maybe maybe it's just that capsule networks are by algorithmic design way way harder to implement like to build specialized hardware for capsule networks I'm not sure if that would even be possible and to speed them up to the degree that CNNs are sped up by GPUs just out of the algorithmic nature of capsule networks and I've done videos on capsule networks they sound pretty cool but they also sound like implementing the thing in hardware is going to be quite tough even if you build specialized hardware they also go into GPT-3 claiming that so current the paper claims that because we are kind of locked in in this neural network this neural network paradigm in this kind of hardware several major research labs are making this bet engaging in a bigger is better race in the number of model parameters and collecting ever more expansive datasets however it is unclear whether this is sustainable an algorithm scalability is often thought of as the performance gradient relative to the available resources given more resources how does the performance increase and they go into examples here that you can scale up the parameters which gives you less and less of a of a gain so it's like this diminishing return over time which it brings up GPT-3 which I find interesting because GPT-3 showed in a way okay was in log space but it showed a fairly fairly linear decrease in perplexity so a log linear decreasing perplexity given more parameters which goes a bit against the narrative of the paper and also in terms of this definition up here given more resources how does the performance increase I see the fact that you say well it's 12 billion sorry 12 million dollars to train GPT-3 says right here 12 million dollars to train GPT-3 on the other hand I would say what's the cost of you know building specialized hardware to research alternative research directions by the way we have no idea what alternative research directions work so the only thing we could do is fund all hardware and if we had to fund all hardware for other algorithms then select the ones that are promising then invest more and so on 12 million dollars will get us nowhere which I think is a point the paper is trying to make but from a efficiency perspective given where we are now it's it's actually more viable to build GPT-3 which again I think this is something the paper agrees with but at the same time it tries to make the point that look we are investing more and more and more and we're getting less and less out of it maybe it's time to go a different route in terms of in terms of hardware but that's going to be more and more expensive the more we go into this neural network direction I'm not yeah I'm not sure about this again if you think of this tree the paper basically tries to argue that what GPT-3 is trying to do is it's trying to make a push up here into the next kind of push the frontier on the path that we have gone for a while and the paper is trying to say that had we gone had we imaginarily gone a different path down here a equally hard push in this direct in a direction would maybe yield a better result yes maybe but yeah but the question is is it at what point does it become viable to sort of abandon this entire direction and skip and kind of start there because we would need to do the whole tree thing again and then within the tree the same logic applies it does though make a good comparison to the human brain which works fundamentally different it says while deep neural networks may be scalable it may be prohibitively expensive to do so in a regime of comparable intelligence to humans an apt metaphor is that we appear to be trying to build a ladder to the moon sort of saying that we can't we can't the way at the rate where we scale neural networks right now it's not conceivable that we reach human level intelligence by simply scaling them up which is why we might want to investigate different entirely different directions and why we might want to investigate entirely different hardware choices yeah which you know granted that's correct though I would say transformers aren't in particularly suited to the hardware because they require such huge memories and GPUs traditionally have been rather limited in memories in memory sorry and and transformers still kick ass on these on this hardware even though memory is extremely limited compared to like CPU memory and only now do we see GPU manufacturers focus on on more memory so you can argue from the perspective of the paper and say see because we have neural network hardware now people are building more neural network hardware but also you can say that initially a bad choice was made sort of but researchers still managed to demonstrate transformers would work and now the hardware is developing in this direction which is also a thing the paper argues at some point again I have a I have a hard point parsing out a direct point here I think the paper is more meant to make you sort of think about think about the different points it brings up which is also probably why this video is more of me rambling than anything else so here it says that currently there are some initiatives to build other types of chips other types of hardware and so on but they as well as the last ones they might be not enough because it takes producing a next-generation chip typically costs 30 to 80 million dollars and two to three years to develop and even that is however even investment of this magnitude may still be woefully inadequate as hardware based on new materials requires long lead times of 10 to 20 years in public investment and is currently far below industry levels of R&D this this is the kind of DARPA and China who funded research in this direction so the paper says it might be way too little though it also says there are a couple of good lights at the end of the tunnel saying experiments using reinforcement learning to optimize chip placement may help decrease cost and I think I've done a video on this paper there are also renewed interest in reconfigurable hardware such as field program gate arrays and coarse-grained reconfigurable configurable arrays so this is hardware that you can sort of metaprogram so you can take the hardware and you can specialize it by programming it and so it's like a metaprogramming it you can sort of take one of these things and make it into like a sort of a GPU if you need it like that and then you can reprogram it program it differently for a different application though if again if I take the other side of this paper I would say well isn't that the same thing that CPUs were and yet still CPUs made it almost impossible for neural networks to run aren't you even though FPGAs are very general aren't you making implicit choices on the ideas that are very well suited to FPGAs or the ideas that are very well suited to using reinforcement learning to optimize chip placement isn't isn't that the exact same thing yeah I guess you can make this argument at in like at infinitum infinitum infinim no infinim is different okay this this video must come must come to an end so the last part here says that what is also needed is kind of a software revolution that there is a shorter feedback time where it imagines software that tells researchers which hardware their algorithm is particularly suited or how their algorithm would fare on different hardware such that if you invent a new algorithm it doesn't work on a GPU you could sort of submit it to this software and then the software will tell you what that this would work really well if type X of hardware existed and then you can maybe invest money into into that rather than discarding your idea in conclusion yeah it doesn't the conclusion isn't very long the performance of an algorithm is fundamentally intertwined with the hardware and software it runs on this essay proposes to term hardware lottery to describe how these downstream choices determine whether a research idea succeeds or fails today the hardware landscape is increasingly heterogeneous this essay posits that the hardware lottery has not gone away and the gap between the winners and losers will grow increasingly larger in order to avoid future hardware lotteries we need to make it easier to quantify the opportunity cost of settling for the hardware and software we have and my conclusion is I generally agree with this paper I really appreciate the the historic overview but I do think the focus is it centers too much around hardware where I think this lottery case you can make for literally any single branching choice and maybe you weigh that by the cost that it takes to revert or change that choice in the future and it also focuses a lot on neural networks versus non neural networks where it kind of yeah this this winners and losers thing where it says neural networks are the winners and if we investigate more into neural networks then they will remain the winners because of this feedback loop however it's kind of in my opinion discards the thing that within the neural networks in the next choice of hardware there are going to be winners and losers again and again and again and they're going to be entire branches of neural network research that are abandoned because they don't fit the hardware choices once more and this gap between what it's conceived the winners and the losers it only it compares losers in terms of an idea that was had in one year to the winners which are always reevaluated every year so it's kind of not a fair comparison in my opinion and then also no that was it for me yes I I do I do implore you if you are interested in things like this as I said this is more of a storical and opinion piece trying to make some argument and give you some directions to think about which is is pretty cool as a change to a simple bland research paper all right that was it for me again if you're still here waiting for how to win the lottery this is not the video bye bye see you next time
 Hi there. Are you interested in winning the lottery? Then let me tell you this
 video is not for you. This video is not about winning the lottery. Okay? I've done
 enough videos with lottery in the title only for people to be mad at me for not
 telling them how to win the lottery. This is about computer science research
 and very unfortunately the author of this paper has decided to put this word
 in the title. So if you're here because you want to win the lottery, this is not
 for you. It's something completely different. For everyone else today we're
 looking at the hardware lottery by Sarah Hooker of Google Brain. This paper is
 it's kind of a mix. It's part of a historic look back at hardware and
 software developments in machine learning and it is a analysis of kind of
 the current situation and an outlook and sort of an opinion piece of the way
 forward and how hardware and software should mix and what we should focus on
 in the future. So the basic the basic principle is quite simple in this paper.
 It introduces this term the hardware lottery. This essay introduces the term
 hardware lottery to describe when a research idea wins because it is compatible
 with available software and hardware and not because the idea is superior to
 alternative research directions. So right off the bat I think this is a
 statement where I think many people can agree or I think almost everyone will
 some agree with this statement in to to a certain degree but certainly to a
 high degree right we are all aware that of course we have the hardware we have
 hardware is very inflexible it's expensive to develop and so on so any
 sort of software development any algorithmic development may simply
 succeed because it is suited to the hardware that we have. So that was my
 first reaction when I read this paper it's a it's a it's a very gut feeling of
 yes of course this is the case but then the historic analysis is also nice but I
 was wondering what is there a deeper reason to to kind of go into this and
 we are going to see some pros and cons that I think in this paper right here
 where it I'm not exactly entirely sure what specific point is trying to make
 the overarching point I completely agree with the fact that of course what
 hardware is here is important and may lead to certain ideas succeeding but it
 I have I have a trouble with the narrower points and I'm gonna try to
 illustrate this in this paper while also telling you what the paper says. So first
 of all here the term is called the hardware lottery but off the bat you
 already see that it says a research idea wins because it is compatible with
 available software and hardware so the hardware lottery right off the bat is
 connect is means that also the software is there so it's technically the hard
 and software lottery and the bigger the bigger question I would have to someone
 arguing that really the hardware lottery is an important concept to have is why
 what does what distinguishes the hardware lottery let's let's even say
 it's just hardware what distinguishes the hardware lottery from any lottery
 like why can't I say okay there's the X lottery and the X lottery is is any
 circumstance any circumstance is that that surrounds a research idea right
 here we have idea one idea two idea three and they all depend on many
 circumstances and X is one of those circumstances and it just so happens that
 the circumstance in the world favors idea two and a different circumstance
 would actually favor idea one what's so special about hardware other than it's
 more expensive than software right to to to illustrate this further let's say
 okay you have you have hardware and you say well hardware is expensive but then
 again you can sort of build a hierarchy where okay down here there is like ideas
 they depend on software like software frameworks that we have such as
 TensorFlow pytorch these again depend on particular hardware but and you can say
 okay the hardware is much more expensive so we are not as flexible and the
 ideas might just succeed because of the hardware but then you can go even step
 further and say well up here is sort of the consumer if you don't like the
 market term then maybe say the society the end user and so on because the
 hardware ultimately is directed towards what humans in society need and that
 changes over time as well so and and it's it's way more expensive to change
 the needs of human society than to change the hardware so I can just also
 claim okay X is now society so the one particular research idea down here might
 win simply because it is more suited to the current societal needs and that kind
 of carries over and you might say well make doesn't that make it a good idea
 doesn't that make it preferable to idea idea to preferable to idea three over
 here that would just optimize for a different society which leads us to the
 question what does it mean to first what does it mean to win here it just says a
 research idea wins and you might have an idea so I've I have an idea it's not
 clearly defined here but maybe winning means that a lot of researchers actually
 research in that direction and the other question is here and not because the
 idea is superior to alternative research directions and here my question would be
 what does superior mean what does it what does it mean for an idea to be
 superior as I said here certainly if an idea is more incongruent with current
 societal needs you might claim it's superior and someone else might say well
 if societal needs were different than a different research idea might be suited
 better the same way someone could say well if hardware was different than a
 different research idea might be better maybe you can say if hardware was
 different a different research idea might be better suited to the current
 needs of society but then I'm pretty sure I can go two three four levels up
 here again so these these terms are a bit vague I think we can all the again
 the initial the initial sentiment when reading this is absolutely in favor
 right I absolutely agree I don't want to want to trash this I just want to sort
 of I try to think a bit deeper about what is actually said here and this is
 where sort of my my troubles start so let's dig a bit into the historic part
 and I think the point the paper is sort of trying to make is that not yet that
 there are specific hardware choices that were made at one particular point and
 because it's so expensive to change hardware that means that a lot of
 researchers simply go along with whatever ideas work on that particular
 hardware that's available and other research ideas are neglected simply
 because the hardware isn't available which again this is a sentiment that I
 think we can we can all agree with so the first part here the paper is
 in the following sections and this is a been important to keep in mind as a red
 thread because I feel one can get lost in the details of the paper so in the
 first section section two we ask what has incentivized the development of
 software hardware and machine learning research in isolation we need to read
 this first this essay begins by acknowledging a crucial paradox machine
 learning researchers mostly ignore hardware despite the role it plays in
 determining what ideas succeed so the argument is that we we develop ideas
 independent of hardware but also we don't it kind of makes it a double
 double point it says that we think we just think about ideas but the ideas we
 might think about may be shaped by the hardware that's available and if we're
 not aware of that we might not we might not see other ideas as viable so section
 two asks what has incentivized the development of software hardware and
 machine learning research in isolation so where does this come from that we
 don't think about the hardware that's at the end section three considers the
 ramifications of this siloed evaluation with examples of early hardware and
 software lotteries so this is the kind of risk historical look back then today
 the hardware landscape is increasingly heterogeneous this essay posits that the
 hardware lottery has not gone away and the gap between the winners and the
 losers will grow increasingly larger so this is a point that the paper
 basically makes that this hardware lottery has not gone away so right now
 we are in this hardware lottery and it does so specifically with regards to
 saying that chips like GPUs and TPUs and even more specialized chips are
 optimized to neural networks and that's why the whole world sort of over focuses
 on neural networks right now and discards other research ideas and the
 gap between the winners and the losers will grow increasingly larger meaning
 that the research ideas that are seen as in viable now if we develop even more
 hardware into that direct into the direction of neural networks those
 research ideas will become more and more inaccessible to the community then
 lastly sections four to five unpack these arguments so the ones that we've
 just seen section six concludes with some thoughts on what it will take to
 avoid future hardware lotteries all right so section two here is this sort of
 historic look back and it goes from these it the point is here separate
 tribes so the point is that something has made it such that the communities
 the software communities and the hardware communities and the idea let's
 say the idea communities the researchers in AI algorithms let's call them the
 algorithmers they they they don't think that much about each other and it makes
 the case that early machines were super duper specialized early machines were
 single use were not expected to be repurposed for a new task because of the
 cost of electronics and the lack of cross-purpose software so early machines
 early computing machines were just single purpose and so on but that all
 changed when the whole world focused on sort of general purpose CPUs that could
 execute any instructions of course according to Turing machine or von
 Neumann architectures so the point that the paper makes is at some point a shift
 happened the general purpose computer area crystallized in 1969 when an
 opinion piece by young engineer called Gordon Moore appeared in electronics
 magazine with the app title cramming more components onto circuit boards
 that's a cool title so this famously gave rise to Moore's law or predicted you
 could double the amount of transistors on an integrated circuit every two years
 and this sort of held true where people stopped building general like sorry
 people stopped building special-purpose hardware but invested just more and more
 and more into building these general-purpose chips these CPUs that and
 the reason why they stopped making specialized hardware is any specialized
 hardware you build will simply be surpassed by the next generation of CPUs
 so even if you make a specific purpose hardware for some problem you just have
 to wait like one or two of these cycles and ordinary general-purpose CPUs will
 simply have will overtake your specialized hardware and since CPUs
 are general purpose the market for them is naturally huge so this this has made
 it such that what was mainly developed was general-purpose CPUs I think the
 paper wants to make the point though I'm not in exactly sure I think it wants to
 make the point that even though the CPUs might be called general-purpose they
 aren't general-purpose like they have their specific advantages and
 disadvantages and that's going to hurt for example neural networks in the years
 following this so in conclusion to this chapter they say in the absence of any
 lever with which to influence hardware development machine learning researchers
 rationally began to treat hardware as a sunk cost to work around rather than
 something fluid that could be shaped however just because we have abstracted
 away hardware does not mean it has ceased to exist early computer science
 history tells us there are many hardware lotteries where the choice of hardware
 and software has determined which idea succeeded and which fail and the example
 is kind of the Charles Babbage's analytic engine that Charles Babbage
 designed but was something like 50 years earlier or so then parts could even be
 manufactured for this idea to succeed and we know many stories of these people
 being ahead of their time and they have this interesting quote I think somewhere
 from Silicon Valley here being too early is the same as being wrong and this
 paper of course focuses on hardware but to come back the conclusion of this
 chapter is that because of this general purpose area because the entire focus
 was on building general purpose CPUs this has led to people not really having
 integrated thought of hardware software algorithm but treating hardware as this
 thing that can execute any instruction and then the the algorithm comes on top
 of this sort of black box that we can't really change we just have the hardware
 we have yeah which which comes back I'm and again I'm not sure like sure that
 that sure I agree that the entire world focusing on general purpose CPUs has
 some influence but certainly hardware is just expensive to make so you could
 argue that even if this hadn't happened a machine learning researcher wouldn't
 necessarily think about the hardware but they would at least have a choice if
 there were a selection of hardwares right okay so that was the section 2
 section 3 now we really go into the historic evidences and there are kind of
 early historic evidence like this Charles Babbage's machine that he invented
 an early example the analytical machine in 1837 and and no it wasn't even decades
 it was only surface during World War two in the first part of the 20th century
 electronic vacuum tubes were heavily used were heavily used for heavily used
 this is I've not I've noticed a number of typos in in the paper I realized it's
 pre-print if the author is listening I can also I can also make a list but this
 this one just popped out for radio communication and radar during World
 War two these vacuum tubes repurposed to provide the compute power necessary to
 break the German enigma code so it would be long after not only after Charles
 Babbage invented this machine but even after he died that people would would
 sort of re-take and in some parts reinvent his ideas to to build modern
 computers the big example though that the paper makes is what it calls the
 lost decades and this is the story of neural networks coupled with two things
 with an AI winter and a focus on expert systems and maybe also though that's not
 entirely mentioned here a focus on things like SVMs so I think it's widely
 known that the main ingredients for neural networks are very very very old
 so here the paper gives some examples back propagation invented in 63
 reinvented reinvented again and deep convolutional networks paired with back
 propagation by on the car it says however it was only three decades later
 that deep neural networks were widely accepted as a promising research
 direction I think this this sort of the timeline here is this here probably
 refers to around 2010 shortly after that of course Alex net beats image net and
 so on but even earlier a bit earlier people were doing heavy research into
 neural networks and three decades later so this is paired with kind of these
 numbers right here let's say 1970 1980 when these ideas were invented presented
 but computers back then were simply unsuited to the to run neural networks
 here it says the gap between these algorithmic advances and empirical
 successes in large part to to incompatible hardware during the general
 purpose computing areas hardware like CPUs were heavily favored and widely
 available CPUs were good at executing any set of complex instructions but
 occur high memory costs because of the need to cache intermediate results and
 process one instruction at a time this is known as the von neumann bottleneck
 the available compute is restricted by the lone channel between CPU and memory
 along which data has to travel sequentially so the paper goes on and
 says there were some efforts into specialized hardware for neural networks
 but funding was kind of not there and other specialized hardware was more into
 the direction of popular ideas then like prologue and lisp which could do expert
 systems and not necessarily neural networks and only only it would take a
 hardware fluke in the early 2000s a full four decades after the first paper
 about back propagation was published for the insight about massive parallelism to
 be operationalized in a useful way for connectionist deep neural networks a
 graphical processing unit was originally introduced in the 1970s as a specialized
 accelerator for video games and developing graphics yada yada yada GPUs
 were repurposed for an entirely unimagined use case to train deep
 neural networks had one critical advantage over CPUs they were far better
 at parallelizing a set of simple decomposable instructions such as matrix
 multiplications multiples multiplications multiples I don't know so
 the the point here is that the ideas were around for a long time but it would
 take GPUs to make them work and so the the image that the paper builds up I
 think is that you have these you're you're here and you research and then
 you have a decision to make which hardware do I build for the future and
 there are two directions this is direction one and this is direction two
 and let's say for whatever reason direction one is chosen okay then
 because it's so expensive to build different hardware the the world largely
 goes with direction one and builds on top of that okay so that also means that
 all the research ideas that profit from direction one will appear to be much
 more effective that research ideas that would have profited from direction two
 and it sort of says that neural networks are over here and it's sort of the and
 the the let's say the other systems what do we give expert systems let's call
 them expert systems and other types of ideas were over here and they appear to
 work really well until they stopped in progress and then by accident sort of
 this road here was traveled use with GPU so it was not obvious but by accident
 still this was developed and then neural networks could flourish and if it wasn't
 for that fluke if it wasn't for video games basically or animation we would
 have never known that neural networks work as well as they do so again that's
 the point the paper makes and I think we can all agree with that particular point
 but I want to again I want to build up sort of a different picture right here
 in that why why is only like I feel hardware is considered a bit much here
 so I think you can make the general case that at any junction you have several
 things you can choose and then once you choose a thing all the things go in that
 direction like new ideas will be more in that direction also new hardware will be
 more in that direction because a lot of people research on it the paper also
 makes the point there's kind of this feedback loop but let's say neural
 networks were down here what I what I would argue and this is a bit of a point
 the paper makes in in a half half formulated way I think is that it
 basically says that had we had we invested in matrix multipliers in GPUs
 instead of CPUs in these early years that means that neural networks would
 have sort of succeeded as an idea at that time and I'm not entirely convinced
 of this because first of all you can see right here GPUs were actually around in
 the 1970s so the hardware was was available it's not it's not like it was
 super easy in in 2010 it was for these early researchers to build their code
 into GPU compatible code that was certainly hard especially if you read
 the papers but it would have been hard in 1970 as well it would not have been
 significantly harder I think so I I'm not sure if the picture is really like
 this or if the picture so if this is the CPU direction is more like that neural
 networks are actually somewhere up here and the fact is we we we actually needed
 the good CPUs in order to develop day in order to make use of the GPUs right and
 this here would be GPU in order to make use of the GPUs to then enable these
 neural networks on the GPUs because certainly it has it has helped a lot
 that CPUs were built that you know computers just built on GPUs would be
 sad computers computers built on CPUs are cool they can do multi-processing
 they can do internet they can do actually they can do most of the video
 game except display the graphics and very arguably that without the heavy
 focus on CPUs we would not have neural networks today even if we had invested
 all of that effort into building GPUs because society has just advanced so
 much because of CPUs so I'm sort of tempted to challenge this notion here
 that just because of the the happenstance that CPUs were advanced at
 that time that neural networks are they didn't have their breakthrough back then
 I think we needed both that being said I do agree with the paper that we might
 have never ever realized that neural networks worked if it weren't for the
 fact that there is specialized hardware around yeah so so that would be my my
 points to this the paper makes yeah makes this point about okay there is
 hardware lotteries and in so now it also introduces software lotteries though it
 said at the beginning that hardware lotteries included software but I'm
 going to guess that the general concept of a lottery was simply presented and
 again I I don't see exactly what's so special about hardware because again I
 can make the same case for software it's just a shorter time frame I can make the
 same case for theory right like whatever now neural tangent kernels are are are
 the hit right everyone's like wow NTKs blah blah blah blah blah who knows right
 but some big names announced this and some theory has been done in this
 direction and because there is already a big momentum lots of people publish in
 it who who knows if that's if that's a good idea if there were other ideas that
 had we done the fundamental work in this would flourish right now they I again I
 don't I agree with the sentiment I don't see why the hardware is the why the
 hardware is is such a special case right here so the next thing that the paper
 looks like it is kind of the current day so it tries to make the point that we
 might be in a hardware lottery right now and again the the intuition of course is
 yes of course we have the hardware we have it's difficult to change especially
 since hardware builds upon hardware with the tree I drew before let's draw it
 again you draw a tree and literally every decision you make in the tree and
 this doesn't only need to be hardware right every single decision you make
 will mean that pretty much all of the previous choices here are now fixed and
 ingrained we build upon we build upon inventions of the past it's impossible
 to go back and do all of these things again and if you see something curious
 right here and this is where we're going to later I want you to see what happens
 if here here is a good idea like here is my super duper booper idea and my super
 duper booper idea simply didn't make the cut for that choice like someone chose a
 different hardware direction software direction software library direction
 whatnot it wasn't in vogue and my idea was unpopular then if one choice is made
 this choice right here it's it's hard to go back if two choices are made right
 that build upon each other it's even harder to go back so as time goes on
 it's harder and harder and harder to go back which is a point that the paper
 will make at the end that the difference between the winners and the losers is
 getting bigger and bigger which is an effect that this idea that once was a
 curiosity that could be investigated becomes a very costly investigation
 because we need to reinvent and re-engineer a whole bunch of decisions
 and it at with time goes on it's simply forgotten because there's so much that
 we have built past this however this is for the loser right this is the loser
 however for the winner I I disagree right here because here it says okay this
 direction the idea direction here let's say there is a super cool idea that
 would beat neural the crap out of neural networks what not whatever whatever the
 latest schmidhuber paper is that that idea would beat neural networks and this
 here is neural networks and everyone's doing neural networks and schmidhuber
 idea is just forgotten about now to say that neural networks are the winner and
 the winners will increase and increase and increase is correct but it forgets
 that right here there is this whole branching so within the neural networks
 you have again this branching and maybe over here what kind of neural networks
 were completely forgotten like MLPs no MLPs are maybe still a thing I don't
 even remember like early early neural networks were 10 H nonlinearities for
 MLPs or something like this 9 by 9 filters 9 by 9 filters in convolution
 things like this right we it's sort of the 9 by 9 filters are technically in
 the class of neural networks but as time progresses and this branch here are the
 3 by 3 filters which are massively out competing the 9 by 9 filters so the 9 by
 9 filters are forgotten and it could be that if the 9 by 9 filters no sorry
 because of the 3 by 3 filters now we have specialized hardware that is
 exclusively focuses on 3 by 3 filters so we go down this route down this route
 down this route down this route and there might have been some other super
 duper idea down here that only works when we have really big filters and now
 we never know that this existed right so they say that the difference between
 the winners and the losers gets bigger and bigger sort of misjudges that these
 winners will be fractionated and fractionated and fractionated and every
 push in one direction comes with costs to these other directions within that
 winner branch but this is I don't yeah ultimately you know you have a choice
 you have a choice do I want to go back and go this direction or do I want to
 add something here it might just might be worth more for society to go up here
 the paper is going to argue at the end that we should sort of keep funding
 alternative directions in hardware which I think is always a good thing to not
 lock in on particular ideas but also you can you sort of have a have to strike a
 balance because you know researching on things that already work and make them
 better is a crucial part as well because you can discard these sub ideas that
 don't make any sense all right so it gives some examples of current hardware
 lottery winners to improve efficiency there is a shift from task agnostic
 hardware like CPUs to domain specialized hardware that tailor the design to make
 certain tasks more efficient the first examples of domain specific hardware at
 least over the last few years TPUs and then it also says edge TPUs Cortec arm
 Cortex m55 Facebook's Big Sur which I think is just like a box with a GPUs in
 it and some Infini band optimize explicitly for costly operations common
 to deep neural networks like matrix multiplies so here I have again there's
 there's this double meaning so it says here is task agnostic hardware like
 CPUs but at the same time it argues that CPUs are particularly bad at matrix
 matrix multiplies it's not really task agnostic it's just focused on on
 different tasks but I see what the what the paper means right here we do build
 hardware that make matrix multiplies faster which means that neural networks
 that benefits neural networks research closer collaboration between hardware
 and research communities will undoubtedly continue to make the training
 and deployment of deep neural networks more efficient for example unstructured
 pruning and weight quantization a very successful compression techniques in
 deep neural networks but in are incompatible with current hardware and
 compilations and compilations kernels hardware and compilations kernels I
 don't know what that means but it's incompatible with current hardware the
 paper argues that because we see that these ideas are good there will be
 specialized hardware for them and I think the point the papers trying to
 make is sort of like see another win for neural networks because we go down the
 neural network road people focus on neural networks focus on how to prune
 them and so on hardware will be developed which will lock us in further
 into neural networks which again is papers basically saying like look
 because we went this road right here we're gonna go this road a lot more but
 then what you have to see is that if we in if we then from this road go here
 because we do want to do weight quantization in this particular way we
 also are going to neglect this which would be doing some whatever other thing
 that we could do yeah so there's always there's always in each decision there's
 a branching undoubtedly the paper is correct and it says the branching
 decides the future but I think the focus here on hardware and neural networks
 versus non neural networks is a bit it's very specific to that thing it then it
 makes the it makes the point why it matters so why it matters it matters
 because the paper says okay where is that here in 2019 the paper was published
 called machine learning is stuck in a rut the authors consider the difficulty
 of training a new type of computer vision architecture called capsule
 networks and I kind of realized that capsule networks aren't really suited to
 current to current to current hardware and he says whether or not you agree
 that capsule networks are the future of computer vision the authors say
 something interesting about the difficulty of trying to train a new type
 of image classification architecture on domain specific specialized hardware
 hardware design has prioritized delivering on commercial use cases while
 built-in flexibility to accommodate the next generation of research ideas
 remains a distant secondary consideration which is true though I
 would also say I mean GPU CPUs and GPUs combined are extremely general
 operations like they're very very generalized okay GPUs are good at matrix
 multiplies but CPUs are good at a lot of other things I would say the GPU CPU
 combo is a very very very flexible general-purpose hardware design that
 doesn't doesn't lock you in too much and maybe maybe it's just that capsule
 networks are by algorithmic design way way harder to implement like to build
 specialized hardware for capsule networks I'm not sure if that would even
 be possible and to speed them up to the degree that CNNs are sped up by GPUs
 just out of the algorithmic nature of capsule networks and I've done videos on
 capsule networks they sound pretty cool but they also sound like implementing
 the thing in hardware is going to be quite tough even if you build specialized
 hardware they also go into GPT-3 claiming that so current the paper claims
 that because we are kind of locked in in this neural network this neural
 network paradigm in this kind of hardware several major research labs are
 making this bet engaging in a bigger is better race in the number of model
 parameters and collecting ever more expansive datasets however it is unclear
 whether this is sustainable an algorithm scalability is often thought of as the
 performance gradient relative to the available resources given more
 resources how does the performance increase and they go into examples here
 that you can scale up the parameters which gives you less and less of a of a
 gain so it's like this diminishing return over time which it brings up GPT-3
 which I find interesting because GPT-3 showed in a way okay was in log space
 but it showed a fairly fairly linear decrease in perplexity so a log linear
 decreasing perplexity given more parameters which goes a bit against the
 narrative of the paper and also in terms of this definition up here given more
 resources how does the performance increase I see the fact that you say
 well it's 12 billion sorry 12 million dollars to train GPT-3 says right here
 12 million dollars to train GPT-3 on the other hand I would say what's the cost
 of you know building specialized hardware to research alternative research
 directions by the way we have no idea what alternative research directions
 work so the only thing we could do is fund all hardware and if we had to fund
 all hardware for other algorithms then select the ones that are promising then
 invest more and so on 12 million dollars will get us nowhere which I think is a
 point the paper is trying to make but from a efficiency perspective given
 where we are now it's it's actually more viable to build GPT-3 which again I
 think this is something the paper agrees with but at the same time it tries to
 make the point that look we are investing more and more and more and we're
 getting less and less out of it maybe it's time to go a different route in
 terms of in terms of hardware but that's going to be more and more expensive the
 more we go into this neural network direction I'm not yeah I'm not sure
 about this again if you think of this tree the paper basically tries to argue
 that what GPT-3 is trying to do is it's trying to make a push up here into the
 next kind of push the frontier on the path that we have gone for a while and
 the paper is trying to say that had we gone had we imaginarily gone a different
 path down here a equally hard push in this direct in a direction would maybe
 yield a better result yes maybe but yeah but the question is is it at what point
 does it become viable to sort of abandon this entire direction and skip and kind
 of start there because we would need to do the whole tree thing again and then
 within the tree the same logic applies it does though make a good comparison to
 the human brain which works fundamentally different it says while
 deep neural networks may be scalable it may be prohibitively expensive to do so
 in a regime of comparable intelligence to humans an apt metaphor is that we
 appear to be trying to build a ladder to the moon sort of saying that we can't
 we can't the way at the rate where we scale neural networks right now it's not
 conceivable that we reach human level intelligence by simply scaling them up
 which is why we might want to investigate different entirely different
 directions and why we might want to investigate entirely different hardware
 choices yeah which you know granted that's correct though I would say
 transformers aren't in particularly suited to the hardware because they
 require such huge memories and GPUs traditionally have been rather limited
 in memories in memory sorry and and transformers still kick ass on these on
 this hardware even though memory is extremely limited compared to like CPU
 memory and only now do we see GPU manufacturers focus on on more memory so
 you can argue from the perspective of the paper and say see because we have
 neural network hardware now people are building more neural network hardware
 but also you can say that initially a bad choice was made sort of but
 researchers still managed to demonstrate transformers would work and now the
 hardware is developing in this direction which is also a thing the paper argues
 at some point again I have a I have a hard point parsing out a direct point
 here I think the paper is more meant to make you sort of think about think about
 the different points it brings up which is also probably why this video is more
 of me rambling than anything else so here it says that currently there are
 some initiatives to build other types of chips other types of hardware and so on
 but they as well as the last ones they might be not enough because it takes
 producing a next-generation chip typically costs 30 to 80 million dollars
 and two to three years to develop and even that is however even investment of
 this magnitude may still be woefully inadequate as hardware based on new
 materials requires long lead times of 10 to 20 years in public investment and is
 currently far below industry levels of R&D this this is the kind of DARPA and
 China who funded research in this direction so the paper says it might be
 way too little though it also says there are a couple of good lights at the end
 of the tunnel saying experiments using reinforcement learning to optimize chip
 placement may help decrease cost and I think I've done a video on this paper
 there are also renewed interest in reconfigurable hardware such as field
 program gate arrays and coarse-grained reconfigurable configurable arrays so
 this is hardware that you can sort of metaprogram so you can take the hardware
 and you can specialize it by programming it and so it's like a metaprogramming it
 you can sort of take one of these things and make it into like a sort of a GPU if
 you need it like that and then you can reprogram it program it differently for
 a different application though if again if I take the other side of this paper I
 would say well isn't that the same thing that CPUs were and yet still CPUs made it
 almost impossible for neural networks to run aren't you even though FPGAs are
 very general aren't you making implicit choices on the ideas that are very well
 suited to FPGAs or the ideas that are very well suited to using reinforcement
 learning to optimize chip placement isn't isn't that the exact same thing
 yeah I guess you can make this argument at in like at infinitum
 infinitum infinim no infinim is different okay this this video must come
 must come to an end so the last part here says that what is also needed is
 kind of a software revolution that there is a shorter feedback time where it
 imagines software that tells researchers which hardware their algorithm is
 particularly suited or how their algorithm would fare on different
 hardware such that if you invent a new algorithm it doesn't work on a GPU you
 could sort of submit it to this software and then the software will tell you what
 that this would work really well if type X of hardware existed and then you can
 maybe invest money into into that rather than discarding your idea in conclusion
 yeah it doesn't the conclusion isn't very long the performance of an algorithm is
 fundamentally intertwined with the hardware and software it runs on this
 essay proposes to term hardware lottery to describe how these downstream choices
 determine whether a research idea succeeds or fails today the hardware
 landscape is increasingly heterogeneous this essay posits that the hardware
 lottery has not gone away and the gap between the winners and losers will grow
 increasingly larger in order to avoid future hardware lotteries we need to make
 it easier to quantify the opportunity cost of settling for the hardware and
 software we have and my conclusion is I generally agree with this paper I really
 appreciate the the historic overview but I do think the focus is it centers too
 much around hardware where I think this lottery case you can make for literally
 any single branching choice and maybe you weigh that by the cost that it takes
 to revert or change that choice in the future and it also focuses a lot on
 neural networks versus non neural networks where it kind of yeah this this
 winners and losers thing where it says neural networks are the winners and if
 we investigate more into neural networks then they will remain the winners
 because of this feedback loop however it's kind of in my opinion discards the
 thing that within the neural networks in the next choice of hardware there are
 going to be winners and losers again and again and again and they're going to be
 entire branches of neural network research that are abandoned because they
 don't fit the hardware choices once more and this gap between what it's conceived
 the winners and the losers it only it compares losers in terms of an idea that
 was had in one year to the winners which are always reevaluated every year so it's
 kind of not a fair comparison in my opinion and then also no that was it for
 me yes I I do I do implore you if you are interested in things like this as I
 said this is more of a storical and opinion piece trying to make some
 argument and give you some directions to think about which is is pretty cool as a
 change to a simple bland research paper all right that was it for me again if
 you're still here waiting for how to win the lottery this is not the video bye
 bye see you next time