All about AI Accelerators: GPU, TPU, Dataflow, Near-Memory, Optical, Neuromorphic & more (w/ Author)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper gpu tpu ipu wave computing dataflow near memory compute ai accelerators deep learning hardware sambanova cerebras graphcore mythic optical computing lightmatter groq why are gpus so fast why does deep learning need gpus do i need a gpu for deep learning transformers hardware hardware matrix multiplication fast deep learning machine learning hardware
#ai #gpu #tpu

This video is an interview with Adi Fuchs, author of a series called "AI Accelerators", and an expert in modern AI acceleration technology.
Accelerators like GPUs and TPUs are an integral part of today's AI landscape. Deep Neural Network training can be sped up by orders of magnitudes by making good use of these specialized pieces of hardware. However, GPUs and TPUs are only the beginning of a vast landscape of emerging technologies and companies that build accelerators for the next generation of AI models. In this interview, we go over many aspects of building hardware for AI, including why GPUs have been so successful, what the most promising approaches look like, how they work, and what the main challenges are.

OUTLINE:
0:00 - Intro
5:10 - What does it mean to make hardware for AI?
8:20 - Why were GPUs so successful?
16:25 - What is "dark silicon"?
20:00 - Beyond GPUs: How can we get even faster AI compute?
28:00 - A look at today's accelerator landscape
30:00 - Systolic Arrays and VLIW
35:30 - Reconfigurable dataflow hardware
40:50 - The failure of Wave Computing
42:30 - What is near-memory compute?
46:50 - Optical and Neuromorphic Computing
49:50 - Hardware as enabler and limiter
55:20 - Everything old is new again
1:00:00 - Where to go to dive deeper?

Read the full blog series here:
Part I: https://medium.com/@adi.fu7/ai-accelerators-part-i-intro-822c2cdb4ca4
Part II: https://medium.com/@adi.fu7/ai-accelerators-part-ii-transistors-and-pizza-or-why-do-we-need-accelerators-75738642fdaa
Part III: https://medium.com/@adi.fu7/ai-accelerators-part-iii-architectural-foundations-3f1f73d61f1f
Part  IV: https://medium.com/@adi.fu7/ai-accelerators-part-iv-the-very-rich-landscape-17481be80917
Part V: https://medium.com/@adi.fu7/ai-accelerators-part-v-final-thoughts-94eae9dbfafb

Links:
Merch: http://store.ykilcher.com
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hello there! Today I'm talking to Adi Fuchs, who is an expert in AI acceleration technology. We talk about a whole bunch of things in this interview, but it is a little bit of a special thing because it's not about a paper or anything, but it is about a series of blog posts that Adi has authored. I am very much a noob in the AI accelerator field, so I thought it'd be really cool to talk to someone who really know what they're talking about, who are in this industry and can explain everything from very technical to very noobish for me. So we go over a whole bunch of things like why do we even need accelerators? What are the reasons behind it? Why are GPUs here and why are they good for AI? Up to very, very modern approaches to AI accelerations, TPUs, and beyond that. So if you're interested in this, watch the interview. It was very cool. I learned a lot and I hope you do too. Without further ado, have fun! Hello everyone! Today I have Adi Fuchs with me right here. He is the author of a series on Medium called AI Accelerators. I have noticed in the last few years and certainly months that I have no clue about hardware. My conception of hardware is something that goes vvvvv, and if I want a neural network, I need a GPU that goes vvvvvv. And then there's TPUs, and then there's IPUs, and there's lots of stuff, but I never had any clue what any of it meant. So this article series was really valuable to me. I thought maybe it's valuable to some of you too. So, Adi, thank you very much for being here. Yeah, thanks for having me and thanks for the kind introduction. Can you tell us a little bit about what your background is in this space? Why did you decide to write a series like this? And why did you think that you had the knowledge to do so? Well, so I've been back and forth between, I would say, industry and academia. I've been working for several hardware and software companies. You know, Philips, I also worked for Mellanox. I also worked for Apple for some, you know, short period. And I've been back and forth. I did my masters back in Israel. And then I did my PhD at the US at the Princeton University. And I always, you know, my studies have been mainly focused on computer architecture. You know, more recently, my experience has been with computer architectures, processor architectures in general. There's a lot of software going on into it. But, you know, from the architectural perspective is how you can design systems that can execute these applications very efficiently. And there's a myriad way of actually doing so. So after my studies, I started working for one of the big companies in the landscape. And I said, actually, when I graduated, I had, when I graduated my PhD, I always had in the back of my mind that AI and machine learning and deep learning, all that has been very, very exciting. You know, I took just like one or two classes, but I didn't really have any extensive experience in it. But I do feel like I do, I was able to see that potential. And I wanted to say, okay, one of the natural things for me after I graduate would be to work for one of those companies that are developing hardware for AI. But, you know, the story goes well beyond just hardware, you know, people right now understand that they need to develop smart systems, smart software, it needs to be a full stack view, just going beyond just like you said, look, the GPU that goes for the TPU or the underlying processor, whatnot. So the landscape seemed to be very exciting. It's rapidly evolving, there are a lot of solutions out there. And I thought that, you know, as a hobby, what I did, it's just started as a hobby, you know, just observing what people are doing, trying to look at the competitive landscape and try to see if there's anything that could be interesting for someone that wants to know more about that world, either be it a research scientist that wants to know a little bit of what's going on under the hood, or people that are hardware engineers that wants to know a little bit more about, you know, the high level motivation for why people are doing AI accelerator. So I was hoping that I will be able to create something like that, that will be able to contribute to several types of people, I would say. Very cool. So my question is a little bit, why, what does it even mean to build hardware for something like obviously, you know, we have computers, and I can, you know, I can do pretty much anything with a computer. What does it mean to, to, to say, make hardware for AI, you have this term of user to hardware expressiveness? What does that mean? So I would say it's, it's, I would, as I said, there is, it's more of my term, in lack of a better term, I would say that probably people have several either academic or industry more accurate ways to depict this is that the user knows on the high level what they're doing, what they want to do, what type of models they want to explore, and how they translate it to high level code, you know, like cafe, pytorch, TensorFlow, and all that. So the research scientist has the big model that they want to explore. But under the hood, there is what the hardware understand it what it can execute. So if you look at it, you can see that there is a lot of layers that you need to go to get you need to lower from the high level code all the way to the, you know, to the bits that are basically executing on on on on, you know, that the electrons that are flowing, and it gets really, really complex, because you need to have a full stack view, and really know, whatever crazy idea that the user is doing, and whatever and, and the last low level detail of everything that your hardware basically can can execute, you know, 80 degrees of parallelism, how it accesses the memory, be it DRAM, high bandwidth memories, HBMs, there's a there's a lot of things that are going on how you're what are your precisions? Are you doing FP 32? Are you doing FP 16, BF 16? Are you doing integers? What is your bit width? And and there are a lot of details that someone needs to understand, in order to build a full flesh, fully capable compiler stack, that you can basically write whatever you can think of, and it'll out of the box be not only working, because as you said, you can basically compute everything right there. I don't know church during thesis, a computer is a computer, but there is a difference between just solving the problem mathematically, or accurately, and actually doing it performant in a performant fashion, because you can either solve a single problem, and it will take a month to run, or you can solve the same problem, and it will be more efficient, it can take, I don't know, like a few hours or even a few minutes. So that's that's the idea of user to hardware expressiveness, you know, the user can think of whatever, and the hardware can execute whatever and you need to bridge that cemented gap between them. And, and, okay, let's say we agree that we need to build hardware for AI, you go through a little bit of the history of that, I guess, starting with what everyone knows, which is kind of Moore's law, that processors or number of transistors increased over time in an exponential fashion, but then you go into some into some less known laws like Dennert scaling, all of this leading up to saying, you know, we've reached the end of clock frequency, I think this is also known. What's also known is probably that the, we have a we have replaced essentially speed with number of cores, and we're going to parallelism. Now you draw an excellent comparison to GPUs here, GPUs being the current super many core architectures, or not current, but in the history, they had more cores. What makes GPUs so attractive for AI in the first place? Yes. So this, I think this goes back a little bit to more of a D intro. You know, you're just saying hardware and you're saying computer, but the fact that you can compute things at certain speeds have been key enablers. I go in the introduction, I'm talking about Alex net, right? You see in the Alex net paper, they say in the abstract, we were able to develop a GPU implementation and efficient GPU implementation that allows it that allowed us to number to crunch a lot of data and train a lot of data within a reasonable timeframe and get a super fancy model that can run efficiently and within reasonable times. And that basically was a key enabler. What I didn't even mention is that for example, for natural language processing, the same story happened. If you look at the attention is all you need paper, they were able to say in the abstract, we were able to train it on GPU for three and a half days, which was order of magnitude pastored and previous solution, you know, all those LSTNs and RNNs that have this inherent sequential part that we were able to devise a new architecture that is able to run on hardware. And just by being able to harness the power of GPUs, we were able to run and it basically unlocked our capabilities. So the ability of hardware has been the role of hardware has been very significant and basically being the key enabler of AI capabilities. And that's why I think this series is more is very important. Going back to our discussion, you know, trying to talk about frequency, it's good to know about the history because when you're talking about AI accelerators is essentially why do we need accelerators? Why and why now? So as you can see, as we said at the beginning, there was frequency, we were able to get our circuitry going faster. You can say that, okay, we have we back at the 90s, you can have like this 486 going at 33 megahertz all the way to like 100 megahertz. Then you came the Pentiums and people will say, yeah, I have like, I don't know, like 300 megahertz and then you go to like a gigahertz. And then ultimately going to the Pentium four with like three or four gigahertz back at the time, you know, during that time, people understood that because you're not able to do the NART scaling, you know, that the NART scaling, what I mentioned there is the actual real problem, you know, going beyond Moore's law, the NART scaling says that it's not only that you can have smaller transistors, they can also go faster and you can cram more transistors and you can have like, if your dimension scales by K, you can have K to the squared number of transistors, each one will be K faster. And the key enabler there was that you were able to, you know, to lower the voltage by that factor. The thing is back at the 2000, the voltage stopped scaling at the rate that you were able to increase the frequency. So you can get faster circuitry, but your power density essentially increases and that's where you can see that the graph that increases and then people say, okay, we cannot have faster transistors. So that was the first stage in the evolution, cannot have faster transistors. You can see like the green dot, the dot is basically plateauing and say, we cannot, so the implication is that we cannot have a single task going faster, but as Moore's law saying, we can still have more transistors. They just cannot go faster. So instead of having one task going fast, we're going to have multiple tasks going at the same speed. So instead of, you know, increasing the frequency twice, we'll have twice the number of cores and depending on how we can map the problem, how efficiently we can map the problem, we'll be able to still get 2X by essentially paralyzing. And that was phase two, which is essentially the multi-core era. So you're able to cram more transistors. They'll be able to getting on the same silicon wafer or the same silicon die. You'll be able to get twice as many cores. And as you can see here, the green line, especially for GPUs as the main beneficent, you're saying, let's develop these instead of having this design, which is the CPU, which has all sorts of very sophisticated mechanisms like stuff that there are branch predictors, prefetchers, and all these speculative things that are saying we can execute an instruction, but this will take too long. We can do out of order execution, but doing all sorts of tricks to make a single stream of instruction go fast. Instead of it, let's do, let's re-devise our software a little bit and break these, the stream of instruction to several independent stream of instructions that are called threads. And we're going to be able to run them hopefully in a perfectly parallel fashion on different, what we call cores and each core will execute its own stream of instructions. So essentially we'll break up one task into multiple subtasks and by that, we'll be able to still get the same degree of speed up. If we'll be able to get it to be able to get like 2X tasks, we'll be able to get a speed up of 2X. Obviously there's a lot of difficulties, but that's the main idea. So we'll be able to, so eventually if we have enough parallelism, we'll be able to get to hundreds or even thousands of cores and we'll be able to get hundreds of thousands of speed up compared to our regular task. But at the mid, I would say the beginning of the 2000, around 2010 and 2011, there were two different works that highlighted the same phenomenon as meaning that because the NART scaling, again, we're not able to scale the voltage, just having transistors powered, not even doing computation, it doesn't matter even at what speed, just having them powered on will increase our power density. Meaning Moore's lie is still working, we can still shrink down the transistors, we can still cram more and more cores into the same silicon square, square millimeter, you know, in the same silicon area, we'll be able to get more transistors to get more cores, but the power at that time will not remain constant. So the power also increases. So that will be unsustainable. And this created the phenomenon that these works are talking about that is called either the utilization wall or dark silicon. Yeah, what's that? That it means that, you know, you can have, let's say a million, it doesn't matter that you're going to have micro transistors, it means that not all cores can be turned on at the same time. Meaning for the purpose of your computation, you're going to remain under a fixed budget, just due to power constraints. So basically what it means that you're not going to be able to get more transistors. And at this point, the power constraints are mainly due to us not being able to cool down a thing that consumes more power. What are the constraints there? So the constraints is that the power density, the watt per millimeter square just starts growing exponentially as you start exponentially cramming more transistors, because the power per transistor stops scaling, it remains constant. So you'll have 1000X transistors, you'll have 1000X to power. And that creates a problem that will be unsus... And that will require cooling that either does not exist or is super expensive to manufacture. So, and that created a problem that essentially says that, okay, we're not going to be able to get more transistors. So if you're not going to be able to get more transistors, then came the notion of building accelerators. Meaning that instead of having a single piece of silicon solving a wide range of problems, you're going to be focused on a little bit of a narrow scope of certain applications. And those applications needs to have some properties. So, and that's the idea. If we're not going to get more transistors, we're going to be able to create smart, purpose-built circuitry with purpose-built compute and memory and communication that is basically targeting specific problems. You can see an example like video encoders, Bitcoin miners, and AI. Yep. So you can see there, if you look at more general purpose processors, if you can look at power efficiency or even performance, you can see that the general purpose processor is fairly does fairly well for a wide application range. But those accelerators are, for example, for FFT or graphs or matrix multiply, they're really good at a certain task, but they do really poorly on something else. For example, you cannot run your operating system or it wouldn't be recommended for you to run your operating system on an AI accelerator. Well, wait, just wait. The community is going to figure it out. You just need to scale enough. But I guess I think from this point on, it's sort of common, let's say common knowledge again, that GPUs were purpose-built for graphics, but inherently that meant kind of matrix multiply things together. And then on the other hand, deep neural networks, just by happenstance, by being ConvNet or feed forward networks, also using a lot of matrix multiplies. And I guess that was just how the universe works. These things came together. And that was just a really neat fit. And the point though is, the GPUs weren't made for AI in the first place, even though it seems to be a really good application for them. GPUs are good for AI, but what can be even better? In which places are GPUs still suboptimal for the AI things that we are doing? Well, it really depends on your application's demands and the application scopes. For example, you can see in the map that you're showing here, you can see that GPUs are really good at flexibility and they're really good in having matrix multiply. As you can say, linear algebra is something that GPUs do pretty well. And if you can map a lot of these problems, like a lot of cons and recommender models and all that, you can map them into a GPU and do dense and to do dense linear algebra pretty well. That will give you a fairly good boost. But if you would devise a certain, you know, if you would go all the way to the efficiency and doing something really, really specialized, you'll be able to say, let's develop an accelerator that just does ResNet, for example. That'll be really, really contrived to collapse to a certain type of network. Theoretically, everything will be hardwired. Even the weights and everything will be perfectly, perfectly fit for that. But it would not be able to execute anything else. So if you would be, yeah, it'll be very, very bad in doing other more general purpose AI. So that comes to question, you know, what, how can you trade flexibility for efficiency? For example, one of the things that some of the companies are, that are not GPU based companies are tackling are these big, these large language models, for example, those GPT-3s and all that. And GPUs, if you look at the A100s, you can see that GPUs from the, I would say that it was a conscious engineering decision for Nvidia to go for high bandwidth models, high bandwidth memories, I'm sorry, that are basically fast memories, but they're limited in capacity. Alternatively, you can go for something else. You can go for a slower DRAM based memory. So HBMs are fast, but they're limited in capacity. And DRAMs are huge and have like terabytes versus, you know, dozens of gigabytes. And if your model requires terabytes of data, you would need hundreds or even thousands of GPUs just to be able to have the same, to do everything in memory, you know, to have the same, to map the memory space of your model. And that would be something that, you know, I'm not saying that GPUs can do, but it would require a lot of GPUs turned on and a lot of power and a lot of communication going on. And so, you know, it would require a lot of communication going from different GPU systems to be able to train a single, you know, like hundreds or hundreds of billions of parameter model. So. I mean, that's exactly what we see, right? Okay. So yeah, I guess we can just dive into what kind of data that goes beyond GPUs exist. That is to say, in part three, okay, in part three of your series, you go into a little bit of the architectural, sorry, foundations, and you describe kind of what, what exists, you know, what instruction sets are, what kind of models exist, for example, configurable processors. You make sort of a good, very extensive background overview, which we're going to skip right now, just due to time. I just found this very, very funny. I guess that's why you posted it here. So there is, this is a single instruction on, that I can use on an Intel processor that computes approximations to the reciprocal square root with less than two to the negative 28, the relative error of the pack double precision floating point values from these things and stores the result in that thing with right mass K one. That is excellent. Like I, I, I need, I need that instruction every day. Yeah. So, you know, depending on the way that, that this is basically showing how you can devise, when you look at a processor, you know, the traditional, the traditional model of processor is called a for Neumann model. It's, you're saying that you're, you have a processor, your processor accesses the memory, your processor fetches an instruction from the memory. It decodes the instruction and says, Oh yeah, we should do this and that. So this instruction accesses the memory and loads, let's fetch the next instruction and all that. So the, the instructions are basically built from an ISA, which is the instruction set architecture, which you can think about it as the vocabulary in which the, the processor says that the processor supports some processors support X86, some processors support arm. And so which, which is, I would say like the X86 is an example of what we call a complex instruction set computing or CISC and arm is the risk. So there was a trade-off between, you know, how much you're going to be able to, to have a single instruction, you know, compact nicely, which will take less memory. So you're going to have a large vocabulary to express more complex computation versus the risk, the reduced instruction set computer like arm that it's going to be basically be translated to a lot of, lot of micro instructions that are B that will be simpler. So that was an ongoing discussion, but you know, this, you know, this gives a background of how basically a processor works. So there are a lot of concepts that I showed at the, at the part three that were basically used as the background for part four, you know, historically I wrote part four as the combination of part three and part four, but someone said, but you know, a lot of people just advised me that this is just going to be super long. So I needed to break it down. So yeah. So if, if anyone, if anyone wants, wants the background, this article is, is really nice on sort of the foundations of all of this. If you, if you want that, and I think people can relate a little bit because in NLP, you have this whole tokenization problem of, you know, how big do you make your vocabulary? And if you make it too small, you're going to have to break down stuff into smaller pieces and so on. Just, I think it's, it's approximately the same concept right here. You're trading essentially memory for, for, for, for speed. And, and also the, the thing is that you need a difficult, you need a very smart compiler to look at your code and say, okay, these sequence of, for example, if you're writing in C, so these sequence of instructions are going to be translated all to that single instruction. And that way you'll have a smart and very, very complex compiler that will be able to map your sequence of operation into that. Sometimes it works and sometimes you're just going to have like these ghost instructions that no one's really going to use. So, So here in part four, I think that that is, it is the longest part. And you dive into the various companies, startups that exist today, building AI, AI accelerators or AI hardware in any form. And it is, we have to say that you are associated with one of those companies. We're not going to say which one though, obviously with the best one. But, but I felt, I felt reading the article that there was no, there was no, I didn't feel any favoritism. So I was, I was pretty happy to see that. Now we have a lot of them even discussed in your articles. Do you maybe have some that you want to, you know, want to highlight in particular to just maybe show the diversity of the field and, and where it's going? Yes. So while there are a lot of solutions out there, I would say most of them stem from a handful of, of, of a few architectural ideas that were highlighted in part three. So I would say that there is originally there's the GPU with the CUDA that has dense linear algebra that is basically has this model, this execution model, single instruction, multiple thread. It's the idea of the classical von Neumann model. You have instructions, they're translated to processor level ISA that the instruction set architecture that Nvidia GPUs understand. And it's being parallelized and it, and you know, it has all these, you know, systolic like execution. And a systolic array is, is an idea that dates back to the 1970s, where you're going to have a single piece of hardware that is really good in doing matrix multiply, because the data, when you're doing matrix multiply, the data from the A and the B matrix is basically flowing like that. And if you have a very smart circuitry like that, which is in a sense, a smart arc accelerator like engine just for matrix multiply, it'll be able to carry out matrix multiply really efficiently. So, yeah, so the GPUs have that. And you can say that there are some other companies that I would say that are in the camp of VLI, a combination of what we call a VLIW, a very large instruction word, where you're going to have a heterogeneous array of compute machines, like a memory compute machine, a vector compute machine, a matrix multiply, and maybe, you know, some sort of a linear compute machine for your re-use or tangents operators and whatnot. Then you have a static compiler that basically creates this huge instruction that says, okay, this data goes to the vector unit, this data goes to the matrix multiply, and this data goes to the vector unit. And you're able to, and you know the timing of all these units, and you'll be able to have a smart compiler that statically creates this single word that is going to be fed to all of them. So you can have, at compile time, a smart compiler that will be able to efficiently schedule these different data or operands to these machines, and they will be able to get really efficient execution. So for, I would say, the systolic slash VLIW camp, I would say things that are, I would, arguably the most famous example is the Google's TPU that was presented at, I would say, mid-2017 at a conference called ISCA, the International Symposium of Computer Architecture, which is the biggest computer architecture conference. So they showed a model that is basically, the TPU is based on a big systolic array execution with a linear unit, and this smart memory, and everything is being fed, and they have a smart compiler that translates AI code for, that is able to execute DNNs, these deep neural nets. And that was the first time, arguably the most famous non-GPU AI accelerator that was presented. So you have the Google TPU. You also have a startup that is called Grok. Some of its founding members were part of the Google TPU team. There were architects at Google that took parts of, that took some of the ideas of Google's TPU and created a more commercialized accelerator for deep neural nets. And also there is Hibana. So I would say Google, Grok, and Hibana are, I would say, the camp VLIW plus systolic array accelerators. So I understand this correctly. Essentially they have a chip or a board, and that has many different, let's say, subchips on it. One is really good at matrix multiplying. One is really good at doing ReLU. One is really good at whatever, softmax. So kind of all these operations that we need in AI, they have like specially subchips for, and then they have a very smart essentially router that says, okay, you go here, you go here, you go here. So, you know, I could compute, let's say, I could compute the last layers ReLU at the same time, or the last batches ReLU at the same time that I compute this layers forward through a linear layer. Is that? Yeah, this is essentially like you're basically pipelining it. So if you have like one thing that needs to ReLU, and then one thing that needs the matrix multiply for the conv operation, then it needs to ReLU, and then you can feed the next sample or whatnot that uses the matrix multiply while the other one is already doing ReLU. So you can do like sort of a pipeline execution. And by that, you're basically filling up your compute machines, right? And by that, you're getting better utilization, because you're using all of your hardware at a single point and everybody's happy and your architecture is perfectly balanced because your compiler is smart enough to understand the program. Yeah. So essentially, we're saying we want the purpose built hardware like the unit that just does ReLU, because that's way better than having a CPU do ReLU. But in order to have the flexibility, we have a bunch of them on a chip and then we have a router and the compiler that knows how to use that router and the pipelines. Okay, excellent. So but that it seems really, it seems like just from for me now, it seems a little bit still in the spirit of like a GPU of what you said that you you essentially have this von Neumann model, except here, there's sort of pipelining added, there is distribution to different subunits added, right, but it's still these kind of instructions that are in sequence and the compiler needs to understand how to translate a program into that. And as I understand the other companies here, they're trying to go sort of bit more out of like out of that paradigm, is that correct? So I would say the, the other big directions that companies are doing is the data flow directions. So some companies are combining two elements, one is called reconfigurability. And the other one is called data flow. So the reconfigurable data flow, I think that tense torrents are doing it, I think that Samba Nova is doing it. Originally, there was a company called wave computing that did it. That are and there is another company, there was another company called simple machines that are doing it. So the idea of reconfigurable data flow is that, first of all, if you look at a pie torch or tensor floor, Keras or a cafe program and AI, a deep learning application, you can see that there are different layers, and they're communicating with each other. So you have a known, a predetermined set of operands, and you know how the data is basically being communicated between different parts of your graph. So in the underlying computation, the data flow, the underlying computation is basically constructing of a computation graph. What does that mean? Like you can see over there, you have your layer. And from that you have another layer that does ReLU, and then you feed it to another conv layer or waste and do that. So you have basically something that is not instruction level, but basically more of the way that your data, you know, you can see that your data is basically flowing between different layers. So the idea is that instead of having that data, that program, that data flow communication graph, go, you flatten it to the classic von Neumann model, then you try to reparalyze it. You can start off from this data flow model, from this data flow graph, and you can basically statically map it via another, again, you need a smart compiler to do that as well. You need to map it to your existing, to a specialized hardware that is capable of executing data flow. Meaning you can have a compute element that does multiply in here, and you can have another one that does add in here, and you can have, you can basically break down your dense linear algebra to compute unit, and you can feed them to other compute unit instead of, you know, breaking down your computation to micro unit, like saying, oh, here's an add, then oh, you need to multiply and all that. So it would be more natural to look at the compute, looking at the computation graph as a data flow graph and map it to the hardware, and you can start it instead of, you know, going back and forth, flattening it to the von Neumann and then parallel, reparalyzing it to the von Neumann. So they're, you know, these companies' bets are that this model is more natural, it's more hardware friendly, and ultimately you can have, you can get a better gain because you're able to have a better, more complex understanding of the graph. You can look at different elements in your graph, you can have a smart compiler that fully understands your hardware, it knows the underline, the number of compute elements and what each compute element in your processor, in your accelerator is doing, and from that it will create a mapping that will essentially go be very static and your data is just going to flow instead of you needing to manually orchestrate it and breaking it down to instructions. So, you know, one of the main selling points of the existing landscape like GPUs is that GPUs are, they have a very mature software stack and they're very flexible, you can program everything from that von Neumann model. If you can create a flexible enough architecture, you'll be able to basically handle new models because, you know, the main challenge for you to build an accelerator company is that it takes two or three years to take out a chip, meaning you need to think about your idea, you need to think about your architecture, all of what you can execute, and you need to be generic enough because within two or three years, it's possible that your application has completely shifted away and if you look at those, the mapping of specialized accelerators, if you're here but your application space is moved here, you're not going to be able to execute it efficiently. So, you need to be very open-minded, you need to be very mindful about being flexible enough to support this. One of the main challenges for that is the ability to create a smart enough software stack that will be able to execute it. So, it's not a trivial task. So, you can take the Wave Computing case as an example. Wave Computing was a company that was really revolutionary. They were able to present a commercialized accelerator that does reconfigurable data flow at the beginning of 2017. So, they had a fancy hardware with 15,000 cores running at 6.7 gigahertz with a lot of engineering complexity that is able to have both slow memory and fast memory and all that. But from what I understood that the CEO interviewed and said, okay, we were not able to succeed in it because it was so complex that going from the basic cases where we were able to showcase a few kernels, trying to generalize that to more complex and real-world application, we found that our hardware software stack had to solve intractable problems and that would become unreasonable. So, I would say that their problem was that they were way, way ahead of the curve. People were just exploring these problems and they were not able to estimate those difficulties. They were pioneers, but ultimately, it didn't pan out so great for them because eventually they filed for bankruptcy. There's also this concept of in-memory compute or near-memory compute. What does that mean? So, there are several notions of how close the compute and your memory should be. One form of near-memory compute is saying that you have your memory model and from that you're loading it to what we call a software control scratchpad memory. So, you have small fast memories. You can think of it as a processor cache, but they're software control. Traditionally, a processor cache like in the Fonoymon model is basically trying, has a heuristic of saving the most recent accesses just because this is the hot data. A software-defined scratchpad memory is something that is more compiler-controlled that you know how you're going to be able to access. One of the guiding principles of devising an accelerator is that you're basically able to anticipate how your memory and data accesses are going to be like. You're going to have a handful of basic, very simple, very simple, very simple, very simple, very simple, very simple basic computational structures that you're going to iterate over a lot of data and it's going to be really recurring. That's one of the things that enable you to develop an accelerator in the first place. So, a scratchpad memory is a very small, a fairly small and fast memory. It can be kilobytes, like a megabyte of data that is really close and it sits within the same piece of, not even the piece of silicon, but within the same core within that piece of silicon and you'll be able to communicate that data fast. It will take like one or two clock cycles. Another approach would be a processor and memory approach. That's when the processing element sits really close to the actual memory model. If you're going to manufacture something like a DRAM or something that is called memristors, which are memory-based resistors, you're going to be able to manufacture a memory module that is going to have logic elements inside of it. You can see of those examples like Mythic or one of those companies that are developing what we call the processor in memory is the idea that you can look at deep learning computation and you can look at the dot product and from that you can do analog computation and that will be fairly, fairly complex. But the idea is that you don't really need to fetch back and forth data from the memory because it's all within this special circuitry that sits within your memory module and you're saving a lot of that energy going back and forth from the memory chip and into a different chip, which is the compute memory, the compute processing element. It's essentially like having a lot of, a lot of cores that we also have lots and lots of registers at those cores, but the registers aren't just for temporary data, but they are actually the memory. In a sense, you can think about it as the difficulty is that you needed to really change the memory that you're manufacturing. And that's something that not a lot of companies are doing, but it's a promising direction because if you have something that is more, that is less depending on your transistors, so it's less prone to the failures of Moore's law. So the end of Moore's law is, might not be the bottleneck for some of these modules, but there are other things like you can see that there's like an analog to digital converter, which could be power hungry and that creates a slew of analog compute problems. There are also a bit more, let's say call them esoteric things that you, all of these were already esoteric to me, but they are, there are more esoteric things like there's like optical computing and neuromorphic computing and things like this. What are, do you have any favorites there or anything that you think is promising and not buzzwordy? I think that these, I think that Lightmatter is a company that is, was founded by a few MIT graduates and they have this idea that light, that representing analog computation via light could be more efficient than using it, but then expressing it through the digital domain. It's an interesting problem. I am not really versed on the different types of difficulties there, but it's sort of like thinking about an analog neuromorphic model where the brain acts basically like on analog pulses. So this is a little bit more trying to mimic the way that the brain works than you would go traditional artificial neural networks where you're going to have a BF16 represent your weights and you can say that this is closer to reality and it's also more energy efficient, but these are, you can say that these are more advanced technologies. So I would say that they probably have their own set of challenges and they're not as efficient as the other challenges. And you never know which one of these technologies will prevail and be the winner. And what is neuromorphic computing? I think that the neuromorphic computing as the way that we know it is the form of analog computing. You're going to have data over here. You're going to have the weights that are sitting within, your memory and your activation is going to be coming from that memory from as inputs to that memory. You're going to be able to do an analog addition and instead of doing that dot product between the weights, you're going to have a single dot product doing vectorized compute in an analog fashion and you're going to be using analog circuitry to compute the results. So it's more of, I would say it's more similar in theory to the spiking neural network model where you're going to have like your brain act on electric pulses. So that's what these solutions are trying to mimic conceptually. And you know that eventually if you look at hardware from the grand scheme of things, you know, you have those accelerators. These accelerators are good at doing AI. But you know, if you really want to get into the definitions, you know, you can go, you can look at the in Goodfellow's deep learning book. It's not really AI. There's an event diagram where AI and inside of it there is machine learning and then there's presentation learning. And then there's deep learning. And from within that deep learning, you can say that these accelerators are good at, you know, a subset of deep learning and a subset of ML that is good at doing matrix multiplication. You know, they're really good at doing things like conv and transformers. But is that a general solution to AI? No one really knows. You know, you can say that the interesting thing is that because the hardware was a key enabler, it's also sort of used as a limiter to what you can achieve. You know, people are saying, is attention all you need? Is conv all you need? Could be. But one thing is for sure is that it consists of most of what your hardware can do. You know, your hardware is really good at transformers and attention and cons. But, you know, is that how intelligence really work? Maybe there's a huge slew of applications that can mimic more human intelligence that are not, that cannot be efficiently ran on hardware accelerators the way that they're built today. And we're not going to be able to explore it just because we don't have the hardware for it and we don't have a way to run it efficiently. So it's an interesting problem. There is this concept, people say this, right, this is a sentiment that's echoed throughout the community that, for example, graph neural networks, we don't have good hardware for graph neural networks, and therefore, probably, we're not going to explore them as much, which also means that hardware manufacturers, since, you know, we can't demonstrate that graph neural networks are really good, won't build graph neural network chips. Do you see this? Do you see it generally going, let's say, more and more converging on some applications? Or do you think, okay, we'll discard some of the applications, but also the ones we have will sort of morph and develop into different variants and so on? Like, how do you see the hardware, essentially the expansiveness of manufacturing hardware's effect on the diversity of the ideas in the field? Do you think there is hope to increase diversity, even with the cost of hardware? It's an interesting question. I would say, obviously, money makes the world go round. If there's money within these applications, you're going to be able to build the hardware for it. The thing is, like we said earlier, hardware has been a key enabler for what you can achieve. And basically, if you cannot run your application on hardware, it will be hard to create that ecosystem for that application to be able to justify building special hardware, because it's a bit of a chicken and an egg problem. If I were to develop an accelerator for a non-Euclidean set of problems, I would first need to look for the applications for it. I will need to be looking for that justification for it, simply because if I'm a startup company, I'm going to have to need funding for it, right? But if you don't have people that are experienced in the industry, you won't be able to find that justification. So it's a bit of a chicken and an egg problem. So as I said, maybe attention is all you need, maybe it's all you need. For surely, it's most of what we have right now. And it would be interesting to see. I would say that, as I said in the final thoughts, I would think that in the next two or three years or so, the things are going to become clearer and architectures are going to be able to stabilize just because we understand the problem better. It will take us four or five years to really converge to a set of common practices and the way that we're developing software libraries and the way that we're developing compilers. We're going to be able to have this I would say three or four stable software stacks that are really good at the conv and transformer games. Will there be other models to create other stacks? Sure. But if I were to start a startup today, it will be really hard for me to go for the conv and the transformers, just because this is a saturated field and people are doing it fairly well and you're basically almost maximizing what you can do in your hardware. The last saying here in your final thoughts is everything old is new again. Do you want to explain what that's about? Yes. It seems like there's a bit of, you can say that on one hand, these models have been the most popular models, those key enablers, those Alex net and those Resnets, those attentions and BERTs and the GPT-3s, they all originated in academic papers, right? But in the hardware field, things are, there's a little bit more of a disconnect. I would say that there are a lot of papers, there are dozens of papers presenting new ideas every year in the top conferences, there are the ESCA, HPCA, ASPLOS and Micro. But eventually you can see that all these fundamental, all these accelerators were basically using ideas originated like 30, 40 years ago. Processing and memories was, I would say in the 1980s, VLIW again, the 1980s, systolic arrays, the 1970s, data flow programming is the 1970s, processing and memory also like 1970s. So it's a bit of conservatism because as you can say that a company building hardware knows, at least in the older days where it was hard to get money funding for it, you would need to really, really justify and really go for these well hashed out ideas before you would go for those wild card ideas. And once you have that, you might be able to explore more revolutionary ideas. Unfortunately, I think that at this point, a lot of your architectural foundations are already established. So you won't be able to explore this crazy accelerators or those things that are really really out there. You'll be able to somewhat integrate it into your existing architecture, but it would be very daring to go and break your entire architecture completely. And especially in a very competitive landscape, you might not be able to go for that risk. You would be surprised, but there are many people in the AI community that say that all the AI ideas have been had in the 80s and 90s as well. And there's essentially nothing new under the sun. But it's a debated position. It's a debated position. Well, I would say that for one thing, for sure, that going back to the attention is all you need and convo is all you need and essentially is what you got. A lot of these, the basic computational structures are already there. People are building on the baseline of these architectures simply because for me as a hardware architect, from my perspective, this is what the hardware can do. It even goes back to this academic notion of accelerators. There's a work called Stream Data Flow Acceleration that was presented in ISCA of 2017, that they're saying, okay, the acceleratable domains need to fulfill certain properties. They need to have a fairly confined control flow. They need to be fairly repetitive. You need to know how the data reuse. You need to know a lot of how your computation patterns behave. So if you're not going to be able to build an accelerator that completely breaks out from this common wisdom and breaks out this template, you might not be able to have an AI model that behaves that way. Is it true or not? Could be or could be not. Maybe we will find out that our existing patterns are fulfilling enough. I would say that there are a lot of problems even within the existing architectures that we were able to fully explore. Cool. Is there anything else you'd like to want to give people on the way? I guess there's not an easy way to necessarily get into hardware yourself at home or something, but if people want to dive, they can certainly go to your articles, which I think are great. I will obviously link them in the video description. Is there any message you want to get out there regarding this? I would say, I cannot really say anything about looking at the blog. Try to look at high level overviews of how hardware and software behaves. It's really tightly coupled today. It's a really exciting time to be either in AI or in hardware because it's a really great opportunity from many aspects historically that you can explore AI hardware either as a research scientist, as a data scientist, or even a computer scientist. It's really good to see how all these pieces pan out. Start looking at the high level overviews and then just deep dive into any of them. Open a computer architecture book. The old ideas are already there. Try to look at the high level white papers from the big companies, the Googles and the NVIDIAs and some of the accelerator companies. Try to understand how your software behaves and you might find that it's not as good as it should be. It's really great that you can execute your models much faster than you have anticipated. If it's going to take you three days to train your model versus if it's going to take you three hours to train your model, it's going to be a key enabler to a lot of your capabilities. Just try to do all those tweaks. Try to understand the common practices. Try to follow programming books and rules and best practices and you might find out that you're going to be able to be a kickass data scientist. Excellent. Well, Adi, it was a great pleasure having you here. I learned a lot. Really, I had no clue before this. Thank you very much for these articles and thanks for being here. Thanks a lot for having me.
 Hello there! Today I'm talking to Adi Fuchs, who is an expert in AI acceleration technology.
 We talk about a whole bunch of things in this interview, but it is a little bit of a special
 thing because it's not about a paper or anything, but it is about a series of blog posts that Adi
 has authored. I am very much a noob in the AI accelerator field, so I thought it'd be really
 cool to talk to someone who really know what they're talking about, who are in this industry
 and can explain everything from very technical to very noobish for me. So we go over a whole bunch
 of things like why do we even need accelerators? What are the reasons behind it? Why are GPUs here
 and why are they good for AI? Up to very, very modern approaches to AI accelerations, TPUs,
 and beyond that. So if you're interested in this, watch the interview. It was very cool. I learned a
 lot and I hope you do too. Without further ado, have fun! Hello everyone! Today I have Adi Fuchs
 with me right here. He is the author of a series on Medium called AI Accelerators. I have noticed in
 the last few years and certainly months that I have no clue about hardware. My conception of hardware
 is something that goes vvvvv, and if I want a neural network, I need a GPU that goes vvvvvv.
 And then there's TPUs, and then there's IPUs, and there's lots of stuff, but I never had any clue
 what any of it meant. So this article series was really valuable to me. I thought maybe it's
 valuable to some of you too. So, Adi, thank you very much for being here.
 Yeah, thanks for having me and thanks for the kind introduction.
 Can you tell us a little bit about what your background is in this space? Why did you decide
 to write a series like this? And why did you think that you had the knowledge to do so?
 Well, so I've been back and forth between, I would say, industry and academia. I've been working for
 several hardware and software companies. You know, Philips, I also worked for Mellanox. I also worked
 for Apple for some, you know, short period. And I've been back and forth. I did my masters back in
 Israel. And then I did my PhD at the US at the Princeton University. And I always, you know,
 my studies have been mainly focused on computer architecture. You know, more recently, my
 experience has been with computer architectures, processor architectures in general. There's a lot
 of software going on into it. But, you know, from the architectural perspective is how you can
 design systems that can execute these applications very efficiently. And there's a myriad way of
 actually doing so. So after my studies, I started working for one of the big companies in the
 landscape. And I said, actually, when I graduated, I had, when I graduated my PhD, I always had in
 the back of my mind that AI and machine learning and deep learning, all that has been very, very
 exciting. You know, I took just like one or two classes, but I didn't really have any extensive
 experience in it. But I do feel like I do, I was able to see that potential. And I wanted to say,
 okay, one of the natural things for me after I graduate would be to work for one of those
 companies that are developing hardware for AI. But, you know, the story goes well beyond just
 hardware, you know, people right now understand that they need to develop smart systems, smart
 software, it needs to be a full stack view, just going beyond just like you said, look, the GPU
 that goes for the TPU or the underlying processor, whatnot. So the landscape seemed to be very
 exciting. It's rapidly evolving, there are a lot of solutions out there. And I thought that,
 you know, as a hobby, what I did, it's just started as a hobby, you know, just observing what people
 are doing, trying to look at the competitive landscape and try to see if there's anything
 that could be interesting for someone that wants to know more about that world, either be it a
 research scientist that wants to know a little bit of what's going on under the hood, or people
 that are hardware engineers that wants to know a little bit more about, you know, the high level
 motivation for why people are doing AI accelerator. So I was hoping that I will be able to create
 something like that, that will be able to contribute to several types of people, I would say.
 Very cool. So my question is a little bit, why, what does it even mean to build hardware for
 something like obviously, you know, we have computers, and I can, you know, I can do pretty
 much anything with a computer. What does it mean to, to, to say, make hardware for AI, you have
 this term of user to hardware expressiveness? What does that mean? So I would say it's, it's,
 I would, as I said, there is, it's more of my term, in lack of a better term, I would say that
 probably people have several either academic or industry more accurate ways to depict this is that
 the user knows on the high level what they're doing, what they want to do, what type of models
 they want to explore, and how they translate it to high level code, you know, like cafe,
 pytorch, TensorFlow, and all that. So the research scientist has the big model that they want to
 explore. But under the hood, there is what the hardware understand it what it can execute.
 So if you look at it, you can see that there is a lot of layers that you need to go to get you need
 to lower from the high level code all the way to the, you know, to the bits that are basically
 executing on on on on, you know, that the electrons that are flowing, and it gets really,
 really complex, because you need to have a full stack view, and really know, whatever crazy idea
 that the user is doing, and whatever and, and the last low level detail of everything that your
 hardware basically can can execute, you know, 80 degrees of parallelism, how it accesses the memory,
 be it DRAM, high bandwidth memories, HBMs, there's a there's a lot of things that are going on how
 you're what are your precisions? Are you doing FP 32? Are you doing FP 16, BF 16? Are you doing
 integers? What is your bit width? And and there are a lot of details that someone needs to understand,
 in order to build a full flesh, fully capable compiler stack, that you can basically write
 whatever you can think of, and it'll out of the box be not only working, because as you said,
 you can basically compute everything right there. I don't know church during thesis,
 a computer is a computer, but there is a difference between just solving the problem mathematically,
 or accurately, and actually doing it performant in a performant fashion, because you can either
 solve a single problem, and it will take a month to run, or you can solve the same problem, and it
 will be more efficient, it can take, I don't know, like a few hours or even a few minutes. So
 that's that's the idea of user to hardware expressiveness, you know, the user can think
 of whatever, and the hardware can execute whatever and you need to bridge that cemented gap
 between them. And, and, okay, let's say we agree that we need to build hardware for AI, you go
 through a little bit of the history of that, I guess, starting with what everyone knows,
 which is kind of Moore's law, that processors or number of transistors increased over time in an
 exponential fashion, but then you go into some into some less known laws like Dennert scaling,
 all of this leading up to saying, you know, we've reached the end of clock frequency,
 I think this is also known. What's also known is probably that the, we have a
 we have replaced essentially speed with number of cores, and we're going to parallelism. Now
 you draw an excellent comparison to GPUs here, GPUs being the current super many core architectures,
 or not current, but in the history, they had more cores. What makes GPUs so attractive for AI in the
 first place? Yes. So this, I think this goes back a little bit to more of a D intro. You know,
 you're just saying hardware and you're saying computer, but the fact that you can compute
 things at certain speeds have been key enablers. I go in the introduction, I'm talking about Alex
 net, right? You see in the Alex net paper, they say in the abstract, we were able to develop a
 GPU implementation and efficient GPU implementation that allows it that allowed us to number to crunch
 a lot of data and train a lot of data within a reasonable timeframe and get a super fancy model
 that can run efficiently and within reasonable times. And that basically was a key enabler.
 What I didn't even mention is that for example, for natural language processing, the same story
 happened. If you look at the attention is all you need paper, they were able to say in the abstract,
 we were able to train it on GPU for three and a half days, which was order of magnitude pastored
 and previous solution, you know, all those LSTNs and RNNs that have this inherent sequential part
 that we were able to devise a new architecture that is able to run on hardware. And just by being
 able to harness the power of GPUs, we were able to run and it basically unlocked our capabilities.
 So the ability of hardware has been the role of hardware has been very significant and basically
 being the key enabler of AI capabilities. And that's why I think this series is more is very
 important. Going back to our discussion, you know, trying to talk about frequency, it's good to know
 about the history because when you're talking about AI accelerators is essentially why do we
 need accelerators? Why and why now? So as you can see, as we said at the beginning, there was
 frequency, we were able to get our circuitry going faster. You can say that, okay, we have we
 back at the 90s, you can have like this 486 going at 33 megahertz all the way to like 100 megahertz.
 Then you came the Pentiums and people will say, yeah, I have like, I don't know, like 300 megahertz
 and then you go to like a gigahertz. And then ultimately going to the Pentium four with like
 three or four gigahertz back at the time, you know, during that time, people understood that
 because you're not able to do the NART scaling, you know, that the NART scaling, what I mentioned
 there is the actual real problem, you know, going beyond Moore's law, the NART scaling says that
 it's not only that you can have smaller transistors, they can also go faster and you can cram more
 transistors and you can have like, if your dimension scales by K, you can have K to the
 squared number of transistors, each one will be K faster. And the key enabler there was that you were
 able to, you know, to lower the voltage by that factor. The thing is back at the 2000, the voltage
 stopped scaling at the rate that you were able to increase the frequency. So you can get faster
 circuitry, but your power density essentially increases and that's where you can see that the
 graph that increases and then people say, okay, we cannot have faster transistors. So that was
 the first stage in the evolution, cannot have faster transistors. You can see like the green
 dot, the dot is basically plateauing and say, we cannot, so the implication is that we cannot
 have a single task going faster, but as Moore's law saying, we can still have more transistors.
 They just cannot go faster. So instead of having one task going fast, we're going to have multiple
 tasks going at the same speed. So instead of, you know, increasing the frequency twice, we'll have
 twice the number of cores and depending on how we can map the problem, how efficiently we can
 map the problem, we'll be able to still get 2X by essentially paralyzing. And that was phase two,
 which is essentially the multi-core era. So you're able to cram more transistors. They'll be able to
 getting on the same silicon wafer or the same silicon die. You'll be able to get twice as many
 cores. And as you can see here, the green line, especially for GPUs as the main beneficent,
 you're saying, let's develop these instead of having this design, which is the CPU, which has
 all sorts of very sophisticated mechanisms like stuff that there are branch predictors,
 prefetchers, and all these speculative things that are saying we can execute an instruction,
 but this will take too long. We can do out of order execution, but doing all sorts of tricks to make
 a single stream of instruction go fast. Instead of it, let's do, let's re-devise our software a
 little bit and break these, the stream of instruction to several independent stream of
 instructions that are called threads. And we're going to be able to run them hopefully in a
 perfectly parallel fashion on different, what we call cores and each core will execute its own
 stream of instructions. So essentially we'll break up one task into multiple subtasks and by that,
 we'll be able to still get the same degree of speed up. If we'll be able to get it to be able
 to get like 2X tasks, we'll be able to get a speed up of 2X. Obviously there's a lot of difficulties,
 but that's the main idea. So we'll be able to, so eventually if we have enough parallelism,
 we'll be able to get to hundreds or even thousands of cores and we'll be able to get
 hundreds of thousands of speed up compared to our regular task. But at the mid, I would say the
 beginning of the 2000, around 2010 and 2011, there were two different works that highlighted
 the same phenomenon as meaning that because the NART scaling, again, we're not able to scale the
 voltage, just having transistors powered, not even doing computation, it doesn't matter even at what
 speed, just having them powered on will increase our power density. Meaning Moore's lie is still
 working, we can still shrink down the transistors, we can still cram more and more cores into the same
 silicon square, square millimeter, you know, in the same silicon area, we'll be able to
 get more transistors to get more cores, but the power at that time will not remain constant.
 So the power also increases. So that will be unsustainable. And this created the phenomenon
 that these works are talking about that is called either the utilization wall or dark silicon.
 Yeah, what's that?
 That it means that, you know, you can have, let's say a million, it doesn't matter that you're going
 to have micro transistors, it means that not all cores can be turned on at the same time.
 Meaning for the purpose of your computation, you're going to remain under a fixed budget,
 just due to power constraints. So basically what it means that you're not going to be able to get
 more transistors. And at this point, the power constraints are mainly due to us not being able
 to cool down a thing that consumes more power. What are the constraints there?
 So the constraints is that the power density, the watt per millimeter square just starts
 growing exponentially as you start exponentially cramming more transistors, because the power per
 transistor stops scaling, it remains constant. So you'll have 1000X transistors, you'll have
 1000X to power. And that creates a problem that will be unsus... And that will require cooling
 that either does not exist or is super expensive to manufacture. So, and that created a problem
 that essentially says that, okay, we're not going to be able to get more transistors.
 So if you're not going to be able to get more transistors, then came the notion of building
 accelerators. Meaning that instead of having a single piece of silicon solving a wide range
 of problems, you're going to be focused on a little bit of a narrow scope of certain applications.
 And those applications needs to have some properties. So, and that's the idea. If we're
 not going to get more transistors, we're going to be able to create smart, purpose-built circuitry
 with purpose-built compute and memory and communication that is basically targeting
 specific problems. You can see an example like video encoders, Bitcoin miners, and AI. Yep.
 So you can see there, if you look at more general purpose processors, if you can look at power
 efficiency or even performance, you can see that the general purpose processor is fairly
 does fairly well for a wide application range. But those accelerators are, for example, for FFT
 or graphs or matrix multiply, they're really good at a certain task, but they do really poorly
 on something else. For example, you cannot run your operating system or it wouldn't be recommended
 for you to run your operating system on an AI accelerator. Well, wait, just wait. The community
 is going to figure it out. You just need to scale enough. But I guess I think from this point on,
 it's sort of common, let's say common knowledge again, that GPUs were purpose-built for graphics,
 but inherently that meant kind of matrix multiply things together. And then on the other hand,
 deep neural networks, just by happenstance, by being ConvNet or feed forward networks, also using
 a lot of matrix multiplies. And I guess that was just how the universe works. These things came
 together. And that was just a really neat fit. And the point though is, the GPUs weren't made for AI
 in the first place, even though it seems to be a really good application for them. GPUs are good
 for AI, but what can be even better? In which places are GPUs still suboptimal for the AI things
 that we are doing? Well, it really depends on your application's demands and the application
 scopes. For example, you can see in the map that you're showing here, you can see that GPUs
 are really good at flexibility and they're really good in having matrix multiply. As you can say,
 linear algebra is something that GPUs do pretty well. And if you can map a lot of these problems,
 like a lot of cons and recommender models and all that, you can map them into a GPU and do dense
 and to do dense linear algebra pretty well. That will give you a fairly good boost. But if you
 would devise a certain, you know, if you would go all the way to the efficiency and doing something
 really, really specialized, you'll be able to say, let's develop an accelerator that just does
 ResNet, for example. That'll be really, really contrived to collapse to a certain type of network.
 Theoretically, everything will be hardwired. Even the weights and everything will be perfectly,
 perfectly fit for that. But it would not be able to execute anything else. So if you would be,
 yeah, it'll be very, very bad in doing other more general purpose AI. So that comes to question,
 you know, what, how can you trade flexibility for efficiency? For example, one of the things that
 some of the companies are, that are not GPU based companies are tackling are these big,
 these large language models, for example, those GPT-3s and all that. And GPUs, if you look at the
 A100s, you can see that GPUs from the, I would say that it was a conscious engineering decision
 for Nvidia to go for high bandwidth models, high bandwidth memories, I'm sorry, that are basically
 fast memories, but they're limited in capacity. Alternatively, you can go for something else.
 You can go for a slower DRAM based memory. So HBMs are fast, but they're limited in capacity.
 And DRAMs are huge and have like terabytes versus, you know, dozens of gigabytes. And if your model
 requires terabytes of data, you would need hundreds or even thousands of GPUs just to be able to have
 the same, to do everything in memory, you know, to have the same, to map the memory space of your
 model. And that would be something that, you know, I'm not saying that GPUs can do, but it would
 require a lot of GPUs turned on and a lot of power and a lot of communication going on. And so,
 you know, it would require a lot of communication going from different GPU systems to be able to
 train a single, you know, like hundreds or hundreds of billions of parameter model. So.
 I mean, that's exactly what we see, right? Okay. So yeah, I guess we can just dive into what kind
 of data that goes beyond GPUs exist. That is to say, in part three, okay, in part three of your
 series, you go into a little bit of the architectural, sorry, foundations, and you describe kind of what,
 what exists, you know, what instruction sets are, what kind of models exist, for example,
 configurable processors. You make sort of a good, very extensive background overview, which we're
 going to skip right now, just due to time. I just found this very, very funny. I guess that's why
 you posted it here. So there is, this is a single instruction on, that I can use on an Intel processor
 that computes approximations to the reciprocal square root with less than two to the negative
 28, the relative error of the pack double precision floating point values from these things
 and stores the result in that thing with right mass K one. That is excellent. Like I, I, I need,
 I need that instruction every day. Yeah. So, you know, depending on the way that, that this is
 basically showing how you can devise, when you look at a processor, you know, the traditional,
 the traditional model of processor is called a for Neumann model. It's, you're saying that you're,
 you have a processor, your processor accesses the memory, your processor fetches an instruction from
 the memory. It decodes the instruction and says, Oh yeah, we should do this and that. So this
 instruction accesses the memory and loads, let's fetch the next instruction and all that. So the,
 the instructions are basically built from an ISA, which is the instruction set architecture,
 which you can think about it as the vocabulary in which the, the processor says that the processor
 supports some processors support X86, some processors support arm. And so which, which is,
 I would say like the X86 is an example of what we call a complex instruction set computing or CISC
 and arm is the risk. So there was a trade-off between, you know, how much you're going to be
 able to, to have a single instruction, you know, compact nicely, which will take less memory. So
 you're going to have a large vocabulary to express more complex computation versus the risk, the
 reduced instruction set computer like arm that it's going to be basically be translated to a lot of,
 lot of micro instructions that are B that will be simpler. So that was an ongoing discussion, but
 you know, this, you know, this gives a background of how basically a processor works. So there are
 a lot of concepts that I showed at the, at the part three that were basically used as the background
 for part four, you know, historically I wrote part four as the combination of part three and part four,
 but someone said, but you know, a lot of people just advised me that this is just going to be
 super long. So I needed to break it down. So yeah. So if, if anyone, if anyone wants,
 wants the background, this article is, is really nice on sort of the foundations of all of this.
 If you, if you want that, and I think people can relate a little bit because in NLP, you have this
 whole tokenization problem of, you know, how big do you make your vocabulary? And if you make it too
 small, you're going to have to break down stuff into smaller pieces and so on. Just, I think it's,
 it's approximately the same concept right here. You're trading essentially memory for,
 for, for, for speed. And, and also the, the thing is that you need a difficult,
 you need a very smart compiler to look at your code and say, okay, these sequence of,
 for example, if you're writing in C, so these sequence of instructions are going to be translated
 all to that single instruction. And that way you'll have a smart and very, very complex compiler
 that will be able to map your sequence of operation into that. Sometimes it works and sometimes you're
 just going to have like these ghost instructions that no one's really going to use. So,
 So here in part four, I think that that is, it is the longest part. And you dive into the various
 companies, startups that exist today, building AI, AI accelerators or AI hardware in any form.
 And it is, we have to say that you are associated with one of those companies. We're not going to
 say which one though, obviously with the best one. But, but I felt, I felt reading the article
 that there was no, there was no, I didn't feel any favoritism. So I was, I was pretty happy to see
 that. Now we have a lot of them even discussed in your articles. Do you maybe have some that you
 want to, you know, want to highlight in particular to just maybe show the diversity of the field and,
 and where it's going? Yes. So while there are a lot of solutions out there, I would say most of them
 stem from a handful of, of, of a few architectural ideas that were highlighted in part three.
 So I would say that there is originally there's the GPU with the CUDA that has dense linear algebra
 that is basically has this model, this execution model, single instruction, multiple thread.
 It's the idea of the classical von Neumann model. You have instructions, they're translated to
 processor level ISA that the instruction set architecture that Nvidia GPUs understand. And
 it's being parallelized and it, and you know, it has all these, you know, systolic like execution.
 And a systolic array is, is an idea that dates back to the 1970s, where you're going to have a
 single piece of hardware that is really good in doing matrix multiply, because the data,
 when you're doing matrix multiply, the data from the A and the B matrix is basically flowing like
 that. And if you have a very smart circuitry like that, which is in a sense, a smart arc accelerator
 like engine just for matrix multiply, it'll be able to carry out matrix multiply really efficiently.
 So, yeah, so the GPUs have that. And you can say that there are some other companies that I would
 say that are in the camp of VLI, a combination of what we call a VLIW, a very large instruction word,
 where you're going to have a heterogeneous array of compute machines, like a memory compute machine,
 a vector compute machine, a matrix multiply, and maybe, you know, some sort of a linear compute
 machine for your re-use or tangents operators and whatnot. Then you have a static compiler that
 basically creates this huge instruction that says, okay, this data goes to the vector unit,
 this data goes to the matrix multiply, and this data goes to the vector unit. And you're able to,
 and you know the timing of all these units, and you'll be able to have a smart compiler that
 statically creates this single word that is going to be fed to all of them. So you can have,
 at compile time, a smart compiler that will be able to efficiently schedule these
 different data or operands to these machines, and they will be able to get really efficient
 execution. So for, I would say, the systolic slash VLIW camp, I would say things that are,
 I would, arguably the most famous example is the Google's TPU that was presented at, I would say,
 mid-2017 at a conference called ISCA, the International Symposium of Computer
 Architecture, which is the biggest computer architecture conference. So they showed a model
 that is basically, the TPU is based on a big systolic array execution with a linear unit,
 and this smart memory, and everything is being fed, and they have a smart compiler that
 translates AI code for, that is able to execute DNNs, these deep neural nets. And that was
 the first time, arguably the most famous non-GPU AI accelerator that was presented.
 So you have the Google TPU. You also have a startup that is called Grok. Some of its
 founding members were part of the Google TPU team. There were architects at Google that
 took parts of, that took some of the ideas of Google's TPU and created a more commercialized
 accelerator for deep neural nets. And also there is Hibana. So I would say Google,
 Grok, and Hibana are, I would say, the camp VLIW plus systolic array accelerators.
 So I understand this correctly. Essentially they have a chip or a board, and that has many
 different, let's say, subchips on it. One is really good at matrix multiplying. One is really good at
 doing ReLU. One is really good at whatever, softmax. So kind of all these operations that we need
 in AI, they have like specially subchips for, and then they have a very smart essentially router
 that says, okay, you go here, you go here, you go here. So, you know, I could compute, let's say,
 I could compute the last layers ReLU at the same time, or the last batches ReLU at the same time
 that I compute this layers forward through a linear layer. Is that? Yeah, this is essentially
 like you're basically pipelining it. So if you have like one thing that needs to ReLU, and then
 one thing that needs the matrix multiply for the conv operation, then it needs to ReLU, and then
 you can feed the next sample or whatnot that uses the matrix multiply while the other one is already
 doing ReLU. So you can do like sort of a pipeline execution. And by that, you're basically filling
 up your compute machines, right? And by that, you're getting better utilization, because you're
 using all of your hardware at a single point and everybody's happy and your architecture is
 perfectly balanced because your compiler is smart enough to understand the program.
 Yeah. So essentially, we're saying we want the purpose built hardware like the unit that just
 does ReLU, because that's way better than having a CPU do ReLU. But in order to have the flexibility,
 we have a bunch of them on a chip and then we have a router and the compiler that knows how to use
 that router and the pipelines. Okay, excellent. So but that it seems really, it seems like just
 from for me now, it seems a little bit still in the spirit of like a GPU of what you said that you
 you essentially have this von Neumann model, except here, there's sort of pipelining added,
 there is distribution to different subunits added, right, but it's still these kind of
 instructions that are in sequence and the compiler needs to understand how to translate
 a program into that. And as I understand the other companies here, they're trying to go sort of
 bit more out of like out of that paradigm, is that correct?
 So I would say the, the other big directions that companies are doing is the data flow directions.
 So some companies are combining two elements, one is called reconfigurability. And the other one is
 called data flow. So the reconfigurable data flow, I think that tense torrents are doing it,
 I think that Samba Nova is doing it. Originally, there was a company called wave computing that
 did it. That are and there is another company, there was another company called simple machines
 that are doing it. So the idea of reconfigurable data flow is that, first of all, if you look at a
 pie torch or tensor floor, Keras or a cafe program and AI, a deep learning application,
 you can see that there are different layers, and they're communicating with each other. So you have
 a known, a predetermined set of operands, and you know how the data is basically being communicated
 between different parts of your graph. So in the underlying computation, the data flow,
 the underlying computation is basically constructing of a computation graph. What does that mean? Like
 you can see over there, you have your layer. And from that you have another layer that does ReLU,
 and then you feed it to another conv layer or waste and do that. So you have basically something
 that is not instruction level, but basically more of the way that your data, you know, you can see
 that your data is basically flowing between different layers. So the idea is that instead of
 having that data, that program, that data flow communication graph, go, you flatten it to the
 classic von Neumann model, then you try to reparalyze it. You can start off from this data flow model,
 from this data flow graph, and you can basically statically map it via another, again, you need a
 smart compiler to do that as well. You need to map it to your existing, to a specialized hardware that
 is capable of executing data flow. Meaning you can have a compute element that does multiply in here,
 and you can have another one that does add in here, and you can have, you can basically break
 down your dense linear algebra to compute unit, and you can feed them to other compute unit instead of,
 you know, breaking down your computation to micro unit, like saying, oh, here's an add, then oh,
 you need to multiply and all that. So it would be more natural to look at the compute, looking at
 the computation graph as a data flow graph and map it to the hardware, and you can start it instead
 of, you know, going back and forth, flattening it to the von Neumann and then parallel, reparalyzing
 it to the von Neumann. So they're, you know, these companies' bets are that this model is more
 natural, it's more hardware friendly, and ultimately you can have, you can get a better gain because
 you're able to have a better, more complex understanding of the graph. You can look at
 different elements in your graph, you can have a smart compiler that fully understands your hardware,
 it knows the underline, the number of compute elements and what each compute element in your
 processor, in your accelerator is doing, and from that it will create a mapping that will essentially
 go be very static and your data is just going to flow instead of you needing to manually orchestrate
 it and breaking it down to instructions. So, you know, one of the main selling points of
 the existing landscape like GPUs is that GPUs are, they have a very mature software stack and they're
 very flexible, you can program everything from that von Neumann model. If you can create
 a flexible enough architecture, you'll be able to basically handle new models because, you know,
 the main challenge for you to build an accelerator company is that it takes two or three years to
 take out a chip, meaning you need to think about your idea, you need to think about your architecture,
 all of what you can execute, and you need to be generic enough because within two or three years,
 it's possible that your application has completely shifted away and if you look at those,
 the mapping of specialized accelerators, if you're here but your application space is moved here,
 you're not going to be able to execute it efficiently. So, you need to be very open-minded,
 you need to be very mindful about being flexible enough to support this. One of the main challenges
 for that is the ability to create a smart enough software stack that will be able to execute it.
 So, it's not a trivial task. So, you can take the Wave Computing case as an example.
 Wave Computing was a company that was really revolutionary. They were able to present a
 commercialized accelerator that does reconfigurable data flow at the beginning of 2017.
 So, they had a fancy hardware with 15,000 cores running at 6.7 gigahertz with
 a lot of engineering complexity that is able to have both slow memory and fast memory and all that.
 But from what I understood that the CEO interviewed and said, okay, we were not able to
 succeed in it because it was so complex that going from the basic cases where we were able to showcase
 a few kernels, trying to generalize that to more complex and real-world application, we found that
 our hardware software stack had to solve intractable problems and that would become
 unreasonable. So, I would say that their problem was that they were
 way, way ahead of the curve. People were just exploring these problems and they were not
 able to estimate those difficulties. They were pioneers, but ultimately, it didn't pan out
 so great for them because eventually they filed for bankruptcy.
 There's also this concept of in-memory compute or near-memory compute. What does that mean?
 So, there are several notions of how close the compute and your memory should be.
 One form of near-memory compute is saying that you have your memory model and from that you're
 loading it to what we call a software control scratchpad memory. So, you have small fast
 memories. You can think of it as a processor cache, but they're software control. Traditionally,
 a processor cache like in the Fonoymon model is basically trying, has a heuristic of saving
 the most recent accesses just because this is the hot data. A software-defined scratchpad memory is
 something that is more compiler-controlled that you know how you're going to be able to access.
 One of the guiding principles of devising an accelerator is that you're basically able to
 anticipate how your memory and data accesses are going to be like. You're going to have a
 handful of basic, very simple, very simple, very simple, very simple, very simple, very simple
 basic computational structures that you're going to iterate over a lot of data and it's going to
 be really recurring. That's one of the things that enable you to develop an accelerator in the first
 place. So, a scratchpad memory is a very small, a fairly small and fast memory. It can be kilobytes,
 like a megabyte of data that is really close and it sits within the same piece of, not even the
 piece of silicon, but within the same core within that piece of silicon and you'll be able to
 communicate that data fast. It will take like one or two clock cycles. Another approach would be
 a processor and memory approach. That's when the processing element sits really close to the actual
 memory model. If you're going to manufacture something like a DRAM or something that is called
 memristors, which are memory-based resistors, you're going to be able to manufacture a
 memory module that is going to have logic elements inside of it. You can see of those examples like
 Mythic or one of those companies that are developing what we call the processor in memory
 is the idea that you can look at deep learning computation and you can look at the dot product
 and from that you can do analog computation and that will be fairly, fairly complex. But the idea
 is that you don't really need to fetch back and forth data from the memory because it's all within
 this special circuitry that sits within your memory module and you're saving a lot of that energy
 going back and forth from the memory chip and into a different chip, which is the compute
 memory, the compute processing element. It's essentially like having a lot of,
 a lot of cores that we also have lots and lots of registers at those cores, but the registers
 aren't just for temporary data, but they are actually the memory. In a sense, you can think
 about it as the difficulty is that you needed to really change the memory that you're manufacturing.
 And that's something that not a lot of companies are doing, but it's a promising direction because
 if you have something that is more, that is less depending on your transistors, so it's less prone
 to the failures of Moore's law. So the end of Moore's law is, might not be the bottleneck for
 some of these modules, but there are other things like you can see that there's like an analog to
 digital converter, which could be power hungry and that creates a slew of analog compute problems.
 There are also a bit more, let's say call them esoteric things that you, all of these were
 already esoteric to me, but they are, there are more esoteric things like there's like optical
 computing and neuromorphic computing and things like this. What are, do you have any favorites
 there or anything that you think is promising and not buzzwordy?
 I think that these, I think that Lightmatter is a company that is, was founded by a few MIT graduates
 and they have this idea that light, that representing analog computation via light
 could be more efficient than using it, but then expressing it through the digital domain.
 It's an interesting problem. I am not really versed on the different types of difficulties there,
 but it's sort of like thinking about an analog neuromorphic model where the brain acts basically
 like on analog pulses. So this is a little bit more trying to mimic the way that the brain works
 than you would go traditional artificial neural networks where you're going to have a BF16
 represent your weights and you can say that this is closer to reality and it's also more energy
 efficient, but these are, you can say that these are more advanced technologies. So I would say
 that they probably have their own set of challenges and they're not as efficient as the
 other challenges. And you never know which one of these technologies will prevail and be the winner.
 And what is neuromorphic computing?
 I think that the neuromorphic computing as the way that we know it is the form of analog computing.
 You're going to have data over here. You're going to have the weights that are sitting within,
 your memory and your activation is going to be coming from that memory from as inputs to that
 memory. You're going to be able to do an analog addition and instead of doing that dot product
 between the weights, you're going to have a single dot product doing vectorized compute in an analog
 fashion and you're going to be using analog circuitry to compute the results. So it's more of,
 I would say it's more similar in theory to the spiking neural network model where you're going
 to have like your brain act on electric pulses. So that's what these solutions are trying to mimic
 conceptually. And you know that eventually if you look at hardware from the grand scheme of things,
 you know, you have those accelerators. These accelerators are good at doing AI. But you know,
 if you really want to get into the definitions, you know, you can go, you can look at the
 in Goodfellow's deep learning book. It's not really AI. There's an event diagram where
 AI and inside of it there is machine learning and then there's presentation learning.
 And then there's deep learning. And from within that deep learning, you can say that these
 accelerators are good at, you know, a subset of deep learning and a subset of ML that is good at
 doing matrix multiplication. You know, they're really good at doing things like conv and
 transformers. But is that a general solution to AI? No one really knows. You know, you can say that
 the interesting thing is that because the hardware was a key enabler, it's also sort of used as a
 limiter to what you can achieve. You know, people are saying, is attention all you need? Is conv all
 you need? Could be. But one thing is for sure is that it consists of most of what your hardware
 can do. You know, your hardware is really good at transformers and attention and cons. But, you
 know, is that how intelligence really work? Maybe there's a huge slew of applications that can
 mimic more human intelligence that are not, that cannot be efficiently ran on hardware accelerators
 the way that they're built today. And we're not going to be able to explore it just because we
 don't have the hardware for it and we don't have a way to run it efficiently. So it's an interesting
 problem.
 There is this concept, people say this, right, this is a sentiment that's echoed throughout the
 community that, for example, graph neural networks, we don't have good hardware for graph neural
 networks, and therefore, probably, we're not going to explore them as much, which also means that
 hardware manufacturers, since, you know, we can't demonstrate that graph neural networks are really
 good, won't build graph neural network chips. Do you see this? Do you see it generally going,
 let's say, more and more converging on some applications? Or do you think, okay, we'll
 discard some of the applications, but also the ones we have will sort of morph and develop into
 different variants and so on? Like, how do you see the hardware, essentially the
 expansiveness of manufacturing hardware's effect on the diversity of the ideas in the field? Do
 you think there is hope to increase diversity, even with the cost of hardware?
 It's an interesting question. I would say, obviously, money makes the world go round. If
 there's money within these applications, you're going to be able to build the hardware for it.
 The thing is, like we said earlier, hardware has been a key enabler for what you can achieve.
 And basically, if you cannot run your application on hardware, it will be hard to create that
 ecosystem for that application to be able to justify building special hardware, because
 it's a bit of a chicken and an egg problem. If I were to develop an accelerator for a
 non-Euclidean set of problems, I would first need to look for the applications for it. I will need
 to be looking for that justification for it, simply because if I'm a startup company, I'm going to
 have to need funding for it, right? But if you don't have people that are experienced in the
 industry, you won't be able to find that justification. So it's a bit of a chicken and
 an egg problem. So as I said, maybe attention is all you need, maybe it's all you need. For
 surely, it's most of what we have right now. And it would be interesting to see. I would say that,
 as I said in the final thoughts, I would think that in the next two or three years or so,
 the things are going to become clearer and architectures are going to be able to stabilize
 just because we understand the problem better. It will take us four or five years to really
 converge to a set of common practices and the way that we're developing software libraries and the
 way that we're developing compilers. We're going to be able to have this
 I would say three or four stable software stacks that are really good at the conv and transformer
 games. Will there be other models to create other stacks? Sure. But if I were to start a startup
 today, it will be really hard for me to go for the conv and the transformers, just because this is
 a saturated field and people are doing it fairly well and you're basically almost maximizing what
 you can do in your hardware. The last saying here in your final thoughts is
 everything old is new again. Do you want to explain what that's about?
 Yes. It seems like there's a bit of, you can say that on one hand, these models have been
 the most popular models, those key enablers, those Alex net and those Resnets, those attentions and
 BERTs and the GPT-3s, they all originated in academic papers, right? But in the hardware field,
 things are, there's a little bit more of a disconnect. I would say that there are a lot of
 papers, there are dozens of papers presenting new ideas every year in the top conferences,
 there are the ESCA, HPCA, ASPLOS and Micro. But eventually you can see that all these fundamental,
 all these accelerators were basically using ideas originated like 30, 40 years ago.
 Processing and memories was, I would say in the 1980s, VLIW again, the 1980s, systolic arrays,
 the 1970s, data flow programming is the 1970s, processing and memory also like 1970s. So it's a
 bit of conservatism because as you can say that a company building hardware knows, at least in the
 older days where it was hard to get money funding for it, you would need to really, really justify
 and really go for these well hashed out ideas before you would go for those wild card ideas.
 And once you have that, you might be able to explore more revolutionary ideas. Unfortunately,
 I think that at this point, a lot of your architectural foundations are already established.
 So you won't be able to explore this crazy accelerators or those things that are really
 really out there. You'll be able to somewhat integrate it into your existing architecture,
 but it would be very daring to go and break your entire architecture completely. And especially in
 a very competitive landscape, you might not be able to go for that risk.
 You would be surprised, but there are many people in the AI community that say that all the AI
 ideas have been had in the 80s and 90s as well. And there's essentially nothing new under the sun.
 But it's a debated position.
 It's a debated position. Well, I would say that for one thing, for sure, that going back to the
 attention is all you need and convo is all you need and essentially is what you got. A lot of these,
 the basic computational structures are already there. People are building on the baseline of
 these architectures simply because for me as a hardware architect, from my perspective,
 this is what the hardware can do. It even goes back to this academic notion of accelerators.
 There's a work called Stream Data Flow Acceleration that was presented in ISCA of 2017,
 that they're saying, okay, the acceleratable domains need to fulfill certain properties.
 They need to have a fairly confined control flow. They need to be fairly repetitive. You need to
 know how the data reuse. You need to know a lot of how your computation patterns behave. So
 if you're not going to be able to build an accelerator that completely breaks out from
 this common wisdom and breaks out this template, you might not be able to have
 an AI model that behaves that way. Is it true or not? Could be or could be not. Maybe we will
 find out that our existing patterns are fulfilling enough. I would say that there are a lot of problems
 even within the existing architectures that we were able to fully explore.
 Cool. Is there anything else you'd like to want to give people on the way? I guess there's not an
 easy way to necessarily get into hardware yourself at home or something, but if people want to dive,
 they can certainly go to your articles, which I think are great. I will obviously link them
 in the video description. Is there any message you want to get out there regarding this?
 I would say, I cannot really say anything about looking at the blog. Try to look at high level
 overviews of how hardware and software behaves. It's really tightly coupled today. It's a really
 exciting time to be either in AI or in hardware because it's a really great opportunity from
 many aspects historically that you can explore AI hardware either as a research scientist,
 as a data scientist, or even a computer scientist. It's really good to see how all these pieces
 pan out. Start looking at the high level overviews and then just deep dive into any of them. Open
 a computer architecture book. The old ideas are already there. Try to look at the high level
 white papers from the big companies, the Googles and the NVIDIAs and some of the accelerator
 companies. Try to understand how your software behaves and you might find that it's not as
 good as it should be. It's really great that you can execute your models much faster than you have
 anticipated. If it's going to take you three days to train your model versus if it's going to take
 you three hours to train your model, it's going to be a key enabler to a lot of your capabilities.
 Just try to do all those tweaks. Try to understand the common practices. Try to follow
 programming books and rules and best practices and you might find out that
 you're going to be able to be a kickass data scientist.
 Excellent. Well, Adi, it was a great pleasure having you here. I learned a lot. Really,
 I had no clue before this. Thank you very much for these articles and thanks for being here.
 Thanks a lot for having me.