How to make your CPU as fast as a GPU - Advances in Sparsity w/ Nir Shavit
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper neuralmagic neural magic deepsparse deep sparse what is deep learning deep learning tutorial introduction to deep learning cpu vs gpu deep learning on cpu deep learning cpu vs gpu
#ai #sparsity #gpu 

Sparsity is awesome, but only recently has it become possible to properly handle sparse models at good performance. Neural Magic does exactly this, using a plain CPU. No specialized hardware needed, just clever algorithms for pruning and forward-propagation of neural networks. Nir Shavit and I talk about how this is possible, what it means in terms of applications, and why sparsity should play a much larger role in the Deep Learning community.

Sponsor: AssemblyAI
Link: https://www.assemblyai.com/?utm_source=youtube&utm_medium=social&utm_campaign=yannic_autochapters

Check out Neural Magic: https://neuralmagic.com/
and DeepSparse: https://github.com/neuralmagic/deepsparse

OUTLINE:
0:00 Introduction
1:08 Sponsor: AssemblyAI
2:50 Start of Interview
4:15 How the NIR company was founded? 
5:10 What is Sparsity about? 
9:30 Link between the human brain and sparsity
12:10 Where should the extra resource that the human brain doesn't have go?
14:40 Analogy for Sparse Architecture
16:48 Possible future for Sparse Architecture as standard architure for Neural Networks
20:08 Pruning & Sparsification
22:57 What keeps us from building sparse models?
25:34 Why are GPUs so unsuited for sparse models?
28:47 CPU and GPU in connection with memory
30:14 What Neural Magic does?
32:54 How do you deal with overlaps in tensor columns?
33:41 The best type of sparsity to execute tons of CPU
37:24 What kind of architecture would make the best use out of a combined system of CPUs and GPUs?
41:04 Graph Neural Networks in connection to sparsity
43:04 Intrinsic connection between the Sparsification of Neural Networks, Non Layer-Wise Computation, Blockchain Technology, Smart Contracts and Distributed Computing
45:23 Neural Magic's target audience
48:16 Is there a type of model where it works particularly well and the type where it doesn't?

Links:
Homepage: https://ykilcher.com
Merch: https://ykilcher.com/merch
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
LinkedIn: https://www.linkedin.com/in/ykilcher

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Today I'm talking to Nir Shavit about sparsity. Nir has been long time active in the field as a professor at Technion and MIT and has also been awarded with various prizes such as the Gödel Prize in 2004 and the Dijkstra Prize in 2012. He's also founder of a company called Neural Magic that questions one of the fundamental core principles of current machine learning, namely, you need GPUs. Neural Magic uses various techniques such as sparsity, which we're going to talk about today, but also other optimization techniques to make inference on models like BERT to be as fast as a GPU on a regular CPU. This is pretty huge and can have vast implications on where you can deploy these models and just how expensive it gets to roll them out to many people in many places. So today we'll talk about the biological foundations for sparsity, why we shouldn't attempt to replicate the brain and just what it takes to make something go really fast on just the CPU. I hope you enjoyed this conversation. If you do give Nir and his company a follow and I'll see you around. Bye bye. Hi, this video is sponsored by assembly AI assembly AI does real time and batch audio transcription of audio and video files powered by the latest advances in artificial intelligence. So if you are a developer or work for a company that's looking to get more out of your audio or video data through transcription and audio intelligence, assembly AI is the best place to go. Not only do they have a user interface where you can just upload stuff, but they do have a very powerful API. But transcription isn't all they do. Once your audio is described, they actually post process it in many different optional ways. So they can do things like speaker classification or annotations of various forms inside of your audio. One feature I'd like to particularly highlight today are the auto chapters for this simply provide auto chapters equals true on your upload and assembly AI will after it's transcribed your audio automatically recognize chunks of audio where you talk about the same thing give you a summary of those chunks and a neat single description headline of what you were talking about there. This is absolutely ideal for anyone who does any sort of long form podcasting or videos like mine, where viewers are very, very helped by the fact that there are chapter annotations and to have these be done automatically is just absolutely great. So if you're interested, head on over to assembly AI use the link in the description to let them know that I sent you there are the single API to transcribe and understand audio, they do so in batch and in real time via web socket, they accept all kinds of audio and video formats. And they do so in over 15 languages, give it a try. And thank you very much to assembly AI for sponsoring this video. And now let's get into the video. The topic of sparsity is a big thing in neural networks right now, mostly because we have no idea really how to do it. And I think that's exciting times for the future. So welcome, what brings you into the sparse world? Actually, I, you know, I've been a professor of computer science for many years, and I worked on multi course for more than 30 years, and got involved in computational neurobiology in the last 10 years. And one of the things that you really see in the brain is really how sparse its computation is. It really is very, very sparse. And so, you know, looking at neural networks, you see that there are there's a similar phenomenon to what happens in brains happening in neural networks, right, where you can actually reduce the number of parameters through pruning by huge amounts and preserve accuracy of the performance of the network. And that kind of says, okay, if we really want to have brain like performance, you know, sparsity is probably one of the tools that we want to use to get there. So that's kind of how I kind of got into this. And you founded a company that also works into this direction, right? You want to talk about that? Yeah, a little bit. Yes. Yes, I founded NeuralMagic. NeuralMagic was founded because what we were seeing in my lab, I was busy with doing machine learning at a large scale for neurobiology projects. And what we realized was that we could get CPUs to run at GPU speeds, like at the time it was a Pascal GPU, and we could make just a regular CPU do what the Pascal GPU was doing through the use of sparsity and other similar techniques. And so we said, okay, well, there's a real commercial value here for people because you don't need an accelerator, you can just do it on your commodity CPU. And that's NeuralMagic. So what we do is we deliver, you know, through sparsity and similar optimization techniques, GPU performance on CPUs. That is quite a promise. Maybe let's first dive into a little bit about sparsity itself. What is it about sparsity? You mentioned the brain is very sparse. Yet our current or at least the way we train neural network is very dense, we can accelerate the dense neural networks much better. What is it about sparsity? Is it just the saving of parameters? Or is there something more to sparse connections than to dense connections? What do we know? That's a good question. So clearly, what we're doing today is not the sparsity that we will be doing in the future. What I mean by that is your brain is sparse way beyond the levels of what we see in neural networks today. So your typical brain in terms of the compute, right, you know, your cortex is like a cell phone of compute, right? But the graph is enormous. It's like, you know, the graph is the size in really petabytes to basically hold it. So a cell phone of compute on a petabyte or more of memory, right? But the accelerators that we build, you know, are designed to deliver petaflops of compute, but on a cell phone size memory. Their memory is very limited because we use this high bandwidth memory. So in a sense, we're building the opposite of what we want, right? So if we want to mimic the brain, we should not busy ourselves so much with the amount of compute and rather worry about how it is that we implement the memory. So we're building this very large graph. It's a very large graph, but it's extremely sparse. That's the point, right? And as you asked, the sparsity is not necessarily the same sparsity that we do today through pruning techniques, but it's a combination of a very sparse architecture together with, you know, a sparsity in what we call in machine learning the kernel, right? So it's not just that the kernels are sparse, but everything in the design is very, very sparse, okay? And we don't know yet how to design very sparse architectures. Part of that has to do with the fact that machine learning grew up in the GPU world where sparsity is not an advantage, actually, because you're doing lockstep computations. So you win nothing by being very sparse. And therefore, you know, we don't see those architectural sparsity things yet, but I'm expecting that to happen. We should be, this should come along, you know? And even more than that, what I expect is things are starting to show up like the pathways from models from Google and so on, where even if you have a very large model, you don't execute the full model layer after layer, but rather you execute small regions of the model at any given time per input. That's another form of sparsification of your computation, right? And that is what the brain really does. So your brain typically, you know, when you see an input or so on, uses a very small fraction of its total graph to do the computation. And so that's where we're headed. We're not there yet. We don't know how to do it. But this is the goal. And that's the old, you only use 10% of the brain at any given time, right? Yeah, that's right. I mean, really from energy considerations, it really is like a cell phone. Okay. It really isn't, you know, this massive monster multi GPU thing that we use today. And so my expectation is that, you know, that as we learn more and more about how to design sparse networks, we're going to see them become the standard. They're not the standard right now, because we started the whole journey, right, by applying flops. And still applying flops is the main paradigm. But we will see it appear both in hardware and accelerators and in CPUs. This idea that we can utilize sparsity, you know, to get really great performance games. Yeah, that's coming. Now, is the question is a little bit the chicken and the egg problem. Is the brain sparse because it has the limitations of the cell phone power? Or does the brain only need cell phone power because sparsity is such a good architecture, right? Like which which causes which? Yeah. So, so I would say that, you know, the whole notion of parallelism in the brain, right? If you think about it, imagine that you need to do a billion operations per second, okay? And what you have are these very slow chemical devices, neurons, right, that can do that, right? So you need a billion operations, a billion, you know, firings of neurons in a second. How are you going to do that? Well, what you need is massive parallelism, right? You've got to get massive parallelism. If you can do the massive parallelism, you can get the billion operations, right? And, and, and so our brains are parallel, if you will, because we have this special medium, right? Now on a modern multiprocessor, right, you can get a billion or 10 billion instructions executed, you know, per second, sequentially, you don't really need parallelism for it, right? And so what I'm trying to say is, you know, the whole idea of, of kind of how brains evolve is clearly because of the way, you know, they're, they're implemented. But we should not think of, of going and implementing this in, in, in silicon in the same way, right? Because we really, what we really should think about just is that both of these things are Turing complete, right? You can do, you can implement the algorithm, you just need to know what the algorithm is. And then on silicon, we'll implement the best algorithm we can, right, you know, of the, of the brain, but we don't have to have the exact architecture of the brain to do that. Okay, does that make sense? That's, that's my, what I'm trying to say here, you know, let's implement the algorithm, but not necessarily the architecture. Okay, so when I say sparsity, I really mean sparsity, algorithmic sparsity, right? And it doesn't mean that you have to have a very sparse kind of, you know, silicon VLSI circuit to do this. That's not the case. Yeah. Given that we, that's a good segue, given that we do have the flops, right, that we don't have in the brain, it naturally, it is a different, a different system, we do have teraflops, petaflops, even in these giant compute clusters, where should we put them, in your opinion, like where, where should that extra resource that the brain doesn't have go? Should it go into sequentially executing what the brain executes in parallel? Or, you know, where should we put that? So first I want to say is that we have those flops, but they're costing us a lot. And you just have to open the papers to see what the cost of the flops is. It's enormous, an enormous energy drain. And it's also an enormous architectural drain on what we're doing. And so I would say, we want to get rid of the flops, because probably we don't need them. Okay. And especially as you go from the data center down to the edge, you get the capability of delivering flops comes directly at the, you know, if at the edge, you can put the, sorry, in the data center, you can put, you know, your Google data warehouse right next to a waterfall or whatever you want, right, to a source of energy, right? When you're doing this on your cell phone or on a tiny device at the edge, every little bit of energy that you waste is critical for you. Right. And so what we really want to do is move away from the flops and move more towards the very energy efficient way the brains work, because this adding more flops is a momentary thing for us. Right. So yes, we can do this, but at a very high cost. And no, we don't want to do this forever. We want to find ways to cut the cost, reduce the compute. And, and, and there's a little other thing that I want to say, and that is architecturally, we generate the flops by running right now, at least by running many, many, many tiny cores, thousands of tiny cores, typically, right. And in architecture, in architectures, they require a lot of connections to the memory, this high bandwidth memory. And this thing doesn't scale. So in a sense, we're trading flops for memory, if you use the CPU today, you could get a terabyte on your desktop, but go get a terabyte on a GPU, right. And so using the flops is going to enable us changing the architecture, if we don't need so many flops, then we can actually increase the size of our memory, which will make us able to hold these giant models that we want to do very cheaply, if you will. If I explain a deep neural network to someone, I usually, you know, you start with a fully connected layer, you say, you know, here is a layer of neurons, and here is a layer of neurons, and they have their connections, right, and each connection has a little weight and so on, you usually describe like a dense, fully connected architecture. And that is conceptually, I want to say, easy to grasp for people and so on. Do you have an analogy for sparse architectures? Like, what is the conceptual like, could you conceptualize to someone who doesn't know what like a sparse architecture is and how to think about it? What is different? Yeah, the way we do sparsity today, I don't know what it will look like in the future. But today, sparsity looks like, imagine that the two layers of the neural network are these kind of, there are cords from one layer to the next, right, there are strings attached, and these are, of course, these are the connections, the weights that we're using in the computation, right. And sparsity means I take scissors, and I chop, chop, chop, chop, chop, you know, till I have five or 10% of those cords left, right. And those cords, it turns out, right, if I do this right, if I do this kind of pruning right, are good enough to capture, right, the accuracy of the model as it was before, because a lot of the connections are not important for this process. That's kind of the big discovery. And modern research in techniques for sparsification, right, you know, play along this kind of game. So you can do this kind of unstructured thing that I just described, where you arbitrarily cut in many places based on the effectiveness, or you can also structurally take things out. So in a lot of the modern models, right, we're removing pieces that are not necessary. We do architecture search to find these places to cut things, right. So that's where the whole game right now of efficiency in neural networks, right, is the game of how do I cut this thing down? Right? In the brain, there are certainly some systems like the visual system, where that is clearly organized into layers. But there are many other systems that have no resemblance to layers, there are connections going up and down and left and right and, you know, between the the halves of the brain and all, is there a possible future where this could become where this could become into like a standard architectures for neural networks that the notion of layers and things like this isn't even really a, you know, a thing anymore? Or is there, you know, some some fundamental way where we say, no, there's probably always going to be layers, but it's just going to be sparsity between those layers. So when we look at, you know, we have a full connectome of essentially only a couple of animals, a worm and a fruit fly, that's it. And that's it. You don't see a lot of layering there. It looks more like a mess, very sparse mess. Okay. And I would, I wouldn't venture to think about how what cortex what a cortex looks like. Right? We don't have that yet. We're working very hard to it's a very, these are very hard computational problems to be able to, to go and get a model, we just want to do a mouse, even a mouse is just too big for us to do right now, like a small mammal. Right. But my, I would venture to guess that yes, the answer is that, you know, it's extremely, it's an extremely sparse architecture, and that it wouldn't, it will not look like layers. Okay. You can impose a layer structure on any graph. Okay. It's not so the idea that I say there aren't layers. Sure. Okay, I can take the graph and I can layer it. Yeah, I could do a BFS on it and layer it. But, but the point is not so much that it's more that by design, when I think about it, right, I'm not going to think about it as a sequence of layers where the change that I make is the change in the layer, one layer is different from the other, but rather, it'll be a combination of thinking about paths, different paths, and I'll do different things along different paths. That's kind of the idea. You know, if you think about, you know, there's recent research from MIT, you know, you can detect, people can detect an image in 0.13, set 0.013 seconds, in 13 milliseconds. Okay. In 13 milliseconds, you can detect it, you can say what an image is. Okay. This is, there's no time for neurons to fire. This thing is extremely kind of parallel, right, and uses very little compute and gets you an answer. And a large part of that is prediction, because you're already expecting something. So we need to learn how to do those things. And so machine learning right now is in a very naive early stage. And so given that and given the things that we are doing right now, it's not, it's not a surprise that we're doing the brute force kind of massive compute kind of thing. That's always what you do. And with time, we're going to get better and better at it. Right. So that's kind of how I see this progressing. Speaking of becoming better, if you know, the flatworm is sparse, the mouse is sparse, the human is certainly sparse. Yet our best models today are all big, dense, you know, computation hungry things, there is not really a case. Every time I prune, I sparseify and so on, I get savings in like, you know, savings in CPU or GPU, I get savings in, you know, my storage, but I also get like a little bit worse, right? That's the common thing today in pruning is that I get like just a tiny bit worse than the dense model I prune from. Why do you do you think that is just the fact that we prune from a dense model? Or what's holding back the sparse models? How about if I if I turn this around? Let me turn this around for you. Okay, you can take you can take BERT base, which is a common model people use, okay. And you can sparsify BERT base. NeuralMagic, we sparsified 95%. So a 95% sparse BERT base, one over 20th of the compute, okay, way beyond anything a GPU does, even if you run it with full throttle, okay, it's just cutting the compute so much that there's really almost nothing to compute there. It's just moving data, okay, not an exaggerating force. But, but you know, it's really becomes a data movement problem rather than a compute problem when you when you and you lose 1% of the compute, you lose 1% less than 1% accuracy. Okay. And I say, Okay, great. So you've done that, you know, and you've gotten all this speed up, but you've lost you say, Oh, near but you lost less than 1% accuracy. But what I say instead is forget that. Take BERT large, a much more accurate model, several points more accurate than BERT base, okay, and prune it so that it actually, right, with 20x less compute, it's actually faster than BERT base. Okay. And so now you have the accuracy, right, and you have great compute, and this is through sparsity. So by sparsifying the larger model, I actually delivered you the best of both worlds, little compute and great accuracy. And that's how I want you to think about sparsity, right. It's a way of enabling us to run much larger, more accurate dense models. But because we sparsified them, we are, you know, we're getting great performance. That's how to think about. What's the limit currently that keeps us from, we always need the dense model first in this model in the pruning in a pruning setup, we first need the dense model, then we go to the sparse model, we get huge savings at inference time, what keeps us from just building the sparse model in the first place? Great. So this is kind of the lottery ticket kind of question, if you will. There is research actually, Dan Alister, one of our consultants at neural magic works exactly on this kind of stuff. We know how to run a training session right now for four models, where you start out and you need to do only a certain fraction of the, you know, of the forward passes, backward passes, dense, and then immediately you can already start pruning while training. So there is research going in that direction. But you are right that right now at least, right in the standard, if you look at what's going on there out there, standardly, you're right. We do most of the time take a standard model and from dense we sparsified and so on. But the thing to remember, and this now I'm not talking about the research, because the research is going to get there. You know, Janek, I don't know if to what extent we will, how fast this will happen and so on, but we will learn how to build sparse architectures, it starts sparse and continues, you know, it's really a matter, nature does this. And so there's no reason why we wouldn't be able to do it. But I want to say something about today's machine learning where you kind of start with the dense and then you have to sparsify. This is really not the common paradigm for most users of neural network. For most users, a model is given to them that, you know, from a known architecture, right? And then they transfer learn onto it. And most people do that rather than train from scratch. They really use the model that somebody already worked very hard to build for their specific use case, and then they transfer learn onto it. So this is what you can do with sparsity. You can take a sparse model and sparse transfer learn onto it. It's extremely efficient because you're running at the speed of the sparse network, right? So you can sparse transfer, and then you don't need all of this kind of start with dense. And we're seeing more and more sparse networks appear in the literature and in the database collections of machine learning models. And as we have more and more of these initial good sparse models, right, people are going to learn to start with the sparse already. That's kind of commercially, I think that's what we're going to see more and more of. Why? You mentioned this a bit already, but why are GPUs so unsuited for sparse models? And what makes CPUs in the way you do it, really suited for sparse models? Or are they even suited? Or are you simply, you know, seeing that they're better? Yeah, I mean, look, the GPU architecture, you know, is designed for this very, you know, small cores, tiny caches. You're not going to go and throw all that away just because, you know, you discovered sparsity. So you're trying to do sparsity while keeping this kind of lockstep execution structure, right? And this is difficult to do sparse. You need really a different kind of setup to get an advantage out of sparsity. Now, I'm not, it's not like you can't do that, right? It's not like you can't do that. People can design and have designed hardware that utilizes sparsity efficiently, okay? There is such hardware. It's just not a, it's not GPU like, it's not like the accelerators that we have today. But all of these, again, all of these accelerators have a different problem that has just to do with the memory. Because of the way they're designed, right, they typically have very small memories. So we're talking, even ones that can run sparse, right, still have the limitation of their memory size. So the reason that CPUs are attractive is not so much that, you know, that they, that you have a natural way of running sparsity because you can run asynchronous with large cores, but rather that the large cores enable you very easy access to very large memory pools, right? So the advantage of having strong, powerful cores, right, is really that I can put several terabytes of memory next to them, right, and run easily. And that's where the big advantage is going to be. As we understand more and more about how to build giant models that don't run all the model layer by layer at the time, right, then the compute will be less important. But actually, the ability to hold that model in one place and run it rather than break it apart on eight or 16 GPUs, that's going to be your advantage. And so this is, so I'm kind of saying it's not so much that you can't build a hard piece of hardware to run sparsity, you can, right? But you should build it looking like a CPU in the sense of you can access a lot of memory because you're not doing tiny cores. That's kind of, that's my two cents. So the CPUs are good because they have, you know, fast connect to large memory, but also over the years, we've put more and more levels of cache onto the CPU. How much do you have to take this into account when you're building, I mean, maybe you can explain a little bit what your company does in terms of software, you build compilers, or can I just run TensorFlow or something? Yeah, so let me explain. So first of all, the connection between the CPU and the memory is slow. GPU has a faster memory and faster access to it, right? Smaller, but fast, right? CPU memory is slow, but large, very large. But CPUs have a cache hierarchy, as you said. And so if you know how to utilize your cache hierarchy, then, you know, if you're running in the L1 cache of a CPU, okay, you're running as fast as the GPU. There's nothing there that the GPU does that the CPU can't do once you're in cache. Okay, in fact, CPU caches are much faster than GPU caches, and the performance is better. So the question then, right, and this is what NeuralMagic does is, okay, so what we do is we sparsify the model. Now, you know, if machine learning is about, okay, I need to meet a certain latency. And because I couldn't meet that latency with a CPU, then we added the GPU and boom, there's machine learning with GPUs. Now I can meet the latency. But there's two ways to deal with latency. One is to add more flops, and the other is to reduce the flops, right? And so sparsity, instead of adding more flops in hardware, reduces the number of flops needed in software. But now that you have this very sparse model, because the CPU memory is slow, okay, then what happens is you hit a bottleneck, and it's very hard to move. If you do this layer after layer, it's very hard to move the data in and out. Okay, so what NeuralMagic invented is a way of running neural networks depth-wise. So we have this technology, which we call tensor columns, where essentially you can run, okay, you know, you can break the model lengthwise and run, you know, each one of these kind of columns, you know, in cache, okay? And because you're not leaving L2 really, or rarely leaving L2, you know, you actually get great performance. So in a sense, right, what we're doing is we're using the natural ability of CPUs to prefetch things from memory and then run in cache. And because this, you know, this cache hierarchy on CPUs has evolved over 70 years, or maybe I'm exaggerating, 60 years of hardware design, it's a very, very well understood thing where people know how to optimize it, right? Especially the big, you know, chip makers, they really know how to make these caches work really well. And so with these really good cache hierarchies, you really get great performance by running the model depth-wise. So that's NeuralMagic, you know, we take the model, sparsify it, now it doesn't need the compute, and now we run it on the CPU and get speed because we're running in cache, okay? And if you look at the numbers, I mean, you know, we are, you know, at the speed of, I mean, some numbers we have in publishing, we're at the speed of an A100, even faster, in terms of how long it takes. A four-core CPU can, in terms of latency, do what a A100 does on a common model at birth, okay? So it's really the... Given that it's sparse or... Yes, yes, yes. By sparsifying it and running it, you can make a four-core do what A100 does. So it's really now a matter of throughput, and the A100 has a lot of throughput, okay? So now the question is, you know, how many cores do you want on your CPU to meet the throughput of the A100? And again, the story is that, you know, the big providers are adding more and more and more cores, so you're going to be able to compete better with the GPUs down the road. So that's kind of the story of NeuralMagic. Yeah. So the way I can imagine these tensor columns is that because I execute depthwise, the sort of values that I need for the next step in the computation are the results of the very last step, therefore, are already going to be in cache. And since everything's sparse, I don't need all of the last layer for the current step, and therefore, you know, I have it already. Right. And of course, when you think about a neural network, there are overlaps between these columns. And the question is, how do you deal with the overlaps in a way that doesn't kill your computation? And that's the magic, right? That's the magic of it. There's an algorithm that allows you to do that. And because you can do it, you manage to run this way, and you don't hit this memory bottleneck, and boom, you're in business. Yeah. So for GPU, it's almost like, you know, GPUs enable us to do dense models. But I think also models have almost co-evolved with the GPUs. So people have started building models to fit the GPU architectures better, right? Especially something like a transformer is like, that's like made for GPUs. Is there a type of model a type of sparse model? Like if you if you could wish for the best possible sparse, but you know, there's different kinds of sparsity, like, what is the best type of sparsity to let's say execute on a CPU? If we want to look forward, and we want to especially build architectures for them? Yeah, this goes back to your original, one of the first questions you asked, right? It's about it's about a different structure for the neural network execution. So we should forget the synchronous layer after layer execution. And think about the fact that, you know, we can run through a model, right? In multiple paths with multiple computing units, use the same weight structure, and so on of the model, right? But run at different speeds. And by running at different speeds, and going through the model in different paths, I can get from the same model, multiple answers to my questions, which is kind of what I believe what your brain does. So what happens there is, you have this network, but it's not like, you know, it's all firing like this layer after layer, it's rather, you have these asynchronous flows going through it, right? Even going through matching paths, and CPUs are naturally built for this thing. Now, I'm not saying that somebody can't build a beautiful FPGA that will perhaps have a better, closer structure to what a brain does. Maybe so, but, you know, but there is an advantage to being commodity. Okay, the fact that the CPU can do other things is a big win. If I can move everything to software is really the thing, is the thing, then I can really get all the advantages of modern software. So I'm not poo-pooing hardware accelerators and saying, great, you know, they have a role and so on and so forth, but they come at a price, right? And the price for any organization is that you, instead of just downloading or shipping your product with the machine learning piece, you have to ask the client to buy a certain accelerator, or run it with a certain accelerator. And this all goes away if we can figure out how to make the CPUs do what the GPUs do, right? Then we have, then we're back into this beautiful world of containerized, movable software. And that's really kind of where I would love machine learning to move to, rather, right? That we would have, and maybe down the road, right? There is this, you know, you know, CPUs have a history of absorbing the key components of any new paradigm that shows up. You know, virtualization started out with tricks on a CPU, and then later on added the features. Networking had special accelerators, and then they moved into the CPU. And I'm expecting that whatever features are necessary for machine learning to run well, will move into the CPU, and we won't need an outside accelerator to make this thing work. If you could. So I think that's, by the way, also the story of GPUs themselves, right? They were already kind of consumerish available. And then they can't they absorbed machine learning. It's not necessarily the best architecture for machine learning. But let's say, let's say there's already all this hardware out there, right? There's very good CPUs next to very good GPUs. How do we get the best out of a machine like this? Right now we've advocated for let's move things to the CPU, right? We have some advantages there. But what if I have a box with both like currently, I just use my CPU to ship data to the GPU, right? That that's what my CPU does. But is there a way where I could potentially, you know, what kind of architecture would make the best use out of a combined system of CPUs and GPUs? No, I think this is really the vision that Nvidia has at least today for their grace Hopper architecture, it's essentially the there will be a CPU and a GPU connected to one another. And the CPU will do all the things that are memory intense, and the GPU will do all the data intent. The thing about the problem with this kind of a model is it's a beautiful model, by the way, I'm not saying anything bad about this. If you if you really want to build a GPU world, that's a great thing to do. But again, the, you know, how you how much you utilize your GPU, your attached GPU has to do with how you write your application, because you need to move the data into the GPU in and out. And that's slow, right? You remember, it's like, it's exactly like going to memory, right? It's the GPU is not up, it's not sitting in your in your caches. So if you're on the CPU, and you're computing something on a cache, and suddenly you get a page fault, and you have to go and get something from memory, that's the latency that the GPU introduces you, right. And so if, if you're going to design it with that, you have to create really good software to pipeline things. And this is at the level of the application. So the application programmer has a big programming task. And so this is a great solution for large scale, big projects where, okay, I'm going to Facebook is going to get, you know, 1000 of these or 10,000 of these, whatever it is, you know, or Google 10,000 100,000 of these and put them together with, then it's worthwhile to write this kind of complex software. But if you're but if you're Joe company, right, and you have your little thing, I don't think you want to be writing that interface, right. So so kind of, so I'm saying it's, it's a it's great for large things, right, data center things, big things. But I'm very doubtful if this is going to be effective at the edge, if you can actually utilize the CPU for it. Okay. And, and I will say one more thing. And that is that, you know, that the modern way that the designers of hardware, think about it is that it's mod, it's built in modules, if you look at the, if you look at the AMD latest architecture, right, essentially, you have the CC axis. So, so the machine, even though it has, you know, maybe 40 or 50 or 60 cores, right, they're grouped into groups of eight, right. And each group of eight like this is a little piece of the die. Okay. And I think Intel is shifting in that direction, too. So nothing's to prevent you from making pieces of that die be specialized pieces of hardware like a GPU, you don't have to have outside device. So if you ask me what the future is going to look like, it's probably going to look like, you know, these large cores, right, that have, or large machines with multiple dies. And on these dies, we might have a GPU die, we might have accelerated. And that's more like what I expect to happen, rather than having a massive, you know, accelerator on the side. If we, if we are sparsity, and things not being in layers, and so on, naturally, the topic of I think graph neural networks is very close to that, at least in the imagination of people, do you have anything to say about, you know, where current graph neural networks stand with respect to sparsity? Yeah, I would think of graph neural networks as a, as a different kind of, okay, so, so graph neural networks, I use some some graph neural networks in my research. And the, and the idea there, you know, is that, you know, we can use graph neural networks to solve graph problems that otherwise would be very complicated to solve if we tried to solve them brute force. Okay, now, it's not generally applicable, there are quite a few limitations. But, but as a tool, I would say that, you know, rather than think about the neural network itself as being looking like a graph neural network, right, I could use graph neural networks, right, to define what we call motifs in the neural network. So for example, when we try to look at, at how brains are structured, right, when we look at the graphs of brains, and we try to understand, you know, is there a motif that is repeating itself in this graph, right, then using a graph neural network for that is a really nice way to try to find these motifs, okay, efficiently, right, because the problem itself is piece-based complete, or we don't know, it's graph isomorphism. So, so clearly, we don't know, right, how to do the brute force algorithm well. But, but the graph neural network can come to our aid here. And so, so I would say that right now, I don't really see a real network design, neural network design that is specific to that, or a way that it helps. But, but in research, it definitely helps. And we really want to use these networks to help us in research. This might be a bit of a tech bro question. But if I hear, you know, I can do sparse computation, very, I can reduce the flops and so on. Is there any intrinsic connection between the sparsification of neural networks, the non layer wise computation, and blockchain technology and smart contracts and distributed computing and things like this? Have you ever given this any thought or or? Yeah, is that completely off? Yeah, look, I think nothing is completely off with respect to machine. That in the sense that I am sure that machine learning will find its way into into all of those areas, right, it's a matter of time. And, and right now, right, the all the work there doesn't need the efficiency of, of, right, of what machine learning offers, because machine learning, in the end, is an optimization technique. And so when I think when all these blockchain algorithms and all, you know, become more common place, and we need to provide them with things like security, further security or analysis, and so on, I think then we're going to see applications of machine learning there. And with that, I think all these things of sparsity and so on, I think are going to appear. But, you know, but but for me, right, it really is the whole story of sparsity, right, is the story of a of a phenomenon that is very prevalent in nature, right, that may you can say, surprisingly or not surprisingly shows up in machine learning. And it kind of it makes me feel like it's strengthening my belief, right, that even though the exact computations that we're doing are not the same as spiking neural networks in brains, right, that there is a lot of commonality there. And the emergence of these similar phenomena, like sparsity, like, you know, pruning and so on, and the fact that we can get benefits from it, this tells me, oh, okay, these are related. I think that's a very important point to keep in mind. With neural magic, who is your main target audience? Like who who is listening to this? Do you want to let know like we are exactly for you? So we span the gamut from the data center to the edge. I would like to say, I mean, we just now are moving into providing the same properties for ARM architectures. And so I would say the exciting new thing in neural magic is we're moving from doing this, you know, for AMD and Intel architectures to doing it for ARM, which means that we're going to span the gamut all the way to the very bottom of the of the food chain, if you will. And I think this is very exciting, because as you know, because because sparsity has a dual role as you go down the food chain, right, because for the large accelerating, you know, the fact that the memory footprint is large is small is not that important. But as I go down, sparsity gives me two things speed with neural magic gives you speed, but it also makes the model extremely small. So you're getting a small, accurate model by running on a very small device. And this, you know, typically is an ARM device. And so that's, that's, that's the audience that I'd like to say, hey, we're coming, you know, we're coming in, we're going to deliver the same things that we can deliver for Intel and AMD, we're now going to deliver it for ARM at the very end. If you say edge, do you mean smartphones? Do you mean security cameras? Do you mean robots? Everything? Okay. I mean, everything. I'm not like I'm going to do everything to start with. But yes, yes, we're aiming in that direction. Yes. And with the danger that this is become going to become like a marketing opportunity question, but how easy is it to get started with what you're doing? Like, let's say I'm, I'm like, I've done, you know, my TensorFlow tutorials, I know how to build a model and train it and so on. Like, how much does it take for me to transition or to apply what you're doing? Yeah, so you just go to our website, go to get go to get download deep sparse, our, you know, our engine download our ML tooling. And, you know, immediately, you just either pick a sparse model and transfer learn onto it with our tool. So we have recipes, you have a model, you have a recipe, exactly what you would do if you went to hugging face and downloaded a model and download a recipe, you do the same kind of thing. And you sparse transfer learn onto it, and you're in business. So it's not very hard. So I think this is really and we're working on making it even even easier. This is one of our goals, right is to make it really, really easy to do this. And the advantage of course, is that, you know, people are already busy, you know, quantizing their models to get more performance. So this is like quantized, in some sense, right, you're going to do the same kind of thing and get a lot more performance. Is there a type of model where it works particularly well and the type of model where it doesn't like I'm thinking, you know, conv nets, recursive networks, autoregressive, maybe, you know, the big language models, like what what is it best at? Yeah, so right now, you know, it's best at at bird YOLO models, we do we do computer vision, and we do, and we do the language models, but not the large language models, we haven't done the large language models yet. So for those types of things like the birds and the YOLOs and the, you know, the whatever the variants of efficient nets and all these guys, this is, you know, visual transformers, these are the things that that we do right now. And and all our technology is right now, you know, available for those, I'd love to do the large models, a CPU is a natural environment for running the knowledge models, you know, these giant models, these trillion or whatever parameter models that people talk about splitting across 16 GPUs, they fit on your desktop. Okay, so clearly, a CPU is a natural place to run a very large model. Okay, and so that's that will be a target, but rotten, but not right now. Okay, very exciting. Is there any last things you want to get out maybe about neural magic or sparsity in general? Well, you know, our our whole machine learning software stack is open source. And we'd love people to come in and help us build, you know, better sparsity use sparsity in their models and, and tell us about what they're doing. And, you know, that it would we have a community, and we'd love you to join us. Excellent. Nier, thank you so much for being here today. This was very pleasant. Thank you very much. Bye bye. Bye bye.
 Today I'm talking to Nir Shavit about sparsity. Nir has been long time active in the field as a
 professor at Technion and MIT and has also been awarded with various prizes such as the Gödel
 Prize in 2004 and the Dijkstra Prize in 2012. He's also founder of a company called Neural Magic that
 questions one of the fundamental core principles of current machine learning, namely, you need GPUs.
 Neural Magic uses various techniques such as sparsity, which we're going to talk about today,
 but also other optimization techniques to make inference on models like BERT to be as fast as a
 GPU on a regular CPU. This is pretty huge and can have vast implications on where you can deploy
 these models and just how expensive it gets to roll them out to many people in many places.
 So today we'll talk about the biological foundations for sparsity, why we shouldn't
 attempt to replicate the brain and just what it takes to make something go really fast on just
 the CPU. I hope you enjoyed this conversation. If you do give Nir and his company a follow and I'll
 see you around. Bye bye. Hi, this video is sponsored by assembly AI assembly AI does real time and batch
 audio transcription of audio and video files powered by the latest advances in artificial intelligence.
 So if you are a developer or work for a company that's looking to get more out of your audio or
 video data through transcription and audio intelligence, assembly AI is the best place to go.
 Not only do they have a user interface where you can just upload stuff, but they do have a very
 powerful API. But transcription isn't all they do. Once your audio is described, they actually
 post process it in many different optional ways. So they can do things like speaker classification
 or annotations of various forms inside of your audio. One feature I'd like to particularly
 highlight today are the auto chapters for this simply provide auto chapters equals true on your
 upload and assembly AI will after it's transcribed your audio automatically recognize chunks of audio
 where you talk about the same thing give you a summary of those chunks and a neat single
 description headline of what you were talking about there. This is absolutely ideal for anyone
 who does any sort of long form podcasting or videos like mine, where viewers are very, very
 helped by the fact that there are chapter annotations and to have these be done automatically is just
 absolutely great. So if you're interested, head on over to assembly AI use the link in the description
 to let them know that I sent you there are the single API to transcribe and understand audio,
 they do so in batch and in real time via web socket, they accept all kinds of audio and video
 formats. And they do so in over 15 languages, give it a try. And thank you very much to assembly AI
 for sponsoring this video. And now let's get into the video. The topic of sparsity is a big thing
 in neural networks right now, mostly because we have no idea really how to do it. And I think
 that's exciting times for the future. So welcome, what brings you into the sparse world? Actually,
 I, you know, I've been a professor of computer science for many years, and I worked on multi
 course for more than 30 years, and got involved in computational neurobiology in the last 10 years.
 And one of the things that you really see in the brain is really how sparse its computation is.
 It really is very, very sparse. And so, you know, looking at neural networks, you see that there are
 there's a similar phenomenon to what happens in brains happening in neural networks, right, where
 you can actually reduce the number of parameters through pruning by huge amounts and preserve
 accuracy of the performance of the network. And that kind of says, okay, if we really want to have
 brain like performance, you know, sparsity is probably one of the tools that we want to use to
 get there. So that's kind of how I kind of got into this. And you founded a company that also
 works into this direction, right? You want to talk about that? Yeah, a little bit. Yes.
 Yes, I founded NeuralMagic. NeuralMagic was founded because what we were seeing in my lab, I was
 busy with doing machine learning at a large scale for neurobiology projects. And what we realized was
 that we could get CPUs to run at GPU speeds, like at the time it was a Pascal GPU, and we could make
 just a regular CPU do what the Pascal GPU was doing through the use of sparsity and other similar
 techniques. And so we said, okay, well, there's a real commercial value here for people because
 you don't need an accelerator, you can just do it on your commodity CPU. And that's NeuralMagic. So
 what we do is we deliver, you know, through sparsity and similar optimization techniques,
 GPU performance on CPUs. That is quite a promise. Maybe let's first dive into a little bit about
 sparsity itself. What is it about sparsity? You mentioned the brain is very sparse.
 Yet our current or at least the way we train neural network is very dense, we can accelerate
 the dense neural networks much better. What is it about sparsity? Is it just the saving of parameters?
 Or is there something more to sparse connections than to dense connections? What do we know?
 That's a good question. So clearly, what we're doing today is not the sparsity that we will be
 doing in the future. What I mean by that is your brain is sparse way beyond the levels of what we
 see in neural networks today. So your typical brain in terms of the compute, right, you know,
 your cortex is like a cell phone of compute, right? But the graph is enormous. It's like,
 you know, the graph is the size in really petabytes to basically hold it. So a cell phone of compute
 on a petabyte or more of memory, right? But the accelerators that we build, you know, are
 designed to deliver petaflops of compute, but on a cell phone size memory. Their memory is very
 limited because we use this high bandwidth memory. So in a sense, we're building the opposite of what
 we want, right? So if we want to mimic the brain, we should not busy ourselves so much with the
 amount of compute and rather worry about how it is that we implement the memory. So we're
 building this very large graph. It's a very large graph, but it's extremely sparse. That's the point,
 right? And as you asked, the sparsity is not necessarily the same sparsity that we do today
 through pruning techniques, but it's a combination of a very sparse architecture
 together with, you know, a sparsity in what we call in machine learning the kernel, right?
 So it's not just that the kernels are sparse, but everything in the design is very, very sparse,
 okay? And we don't know yet how to design very sparse architectures. Part of that has to do with
 the fact that machine learning grew up in the GPU world where sparsity is not an advantage, actually,
 because you're doing lockstep computations. So you win nothing by being very sparse. And therefore,
 you know, we don't see those architectural sparsity things yet, but I'm expecting that
 to happen. We should be, this should come along, you know? And even more than that, what I expect
 is things are starting to show up like the pathways from models from Google and so on, where
 even if you have a very large model, you don't execute the full model layer after layer, but
 rather you execute small regions of the model at any given time per input. That's another form
 of sparsification of your computation, right? And that is what the brain really does. So your brain
 typically, you know, when you see an input or so on, uses a very small fraction of its total graph
 to do the computation. And so that's where we're headed. We're not there yet. We don't know how to
 do it. But this is the goal. And that's the old, you only use 10% of the brain at any given time,
 right? Yeah, that's right. I mean, really from energy considerations, it really is like a cell
 phone. Okay. It really isn't, you know, this massive monster multi GPU thing that we use today.
 And so my expectation is that, you know, that as we learn more and more about how to design
 sparse networks, we're going to see them become the standard. They're not the standard right now,
 because we started the whole journey, right, by applying flops. And still applying flops is the
 main paradigm. But we will see it appear both in hardware and accelerators and in CPUs.
 This idea that we can utilize sparsity, you know, to get really great performance games. Yeah,
 that's coming. Now, is the question is a little bit the chicken and the egg problem. Is the brain
 sparse because it has the limitations of the cell phone power? Or does the brain only need cell phone
 power because sparsity is such a good architecture, right? Like which which causes which?
 Yeah. So, so I would say that, you know, the whole notion of parallelism in the brain, right?
 If you think about it, imagine that you need to do a billion operations per second,
 okay? And what you have are these very slow chemical devices, neurons, right, that can do that,
 right? So you need a billion operations, a billion, you know, firings of neurons in a second. How are
 you going to do that? Well, what you need is massive parallelism, right? You've got to get
 massive parallelism. If you can do the massive parallelism, you can get the billion operations,
 right? And, and, and so our brains are parallel, if you will, because we have this special medium,
 right? Now on a modern multiprocessor, right, you can get a billion or 10 billion instructions
 executed, you know, per second, sequentially, you don't really need parallelism for it, right?
 And so what I'm trying to say is, you know, the whole idea of, of kind of how brains evolve is
 clearly because of the way, you know, they're, they're implemented. But we should not think of,
 of going and implementing this in, in, in silicon in the same way, right? Because we really, what we
 really should think about just is that both of these things are Turing complete, right? You can
 do, you can implement the algorithm, you just need to know what the algorithm is. And then on silicon,
 we'll implement the best algorithm we can, right, you know, of the, of the brain, but we don't have
 to have the exact architecture of the brain to do that. Okay, does that make sense? That's, that's
 my, what I'm trying to say here, you know, let's implement the algorithm, but not necessarily the
 architecture. Okay, so when I say sparsity, I really mean sparsity, algorithmic sparsity, right?
 And it doesn't mean that you have to have a very sparse kind of, you know, silicon VLSI circuit
 to do this. That's not the case. Yeah.
 Given that we, that's a good segue, given that we do have the flops, right, that we don't have in
 the brain, it naturally, it is a different, a different system, we do have teraflops, petaflops,
 even in these giant compute clusters, where should we put them, in your opinion, like where,
 where should that extra resource that the brain doesn't have go? Should it go into sequentially
 executing what the brain executes in parallel? Or, you know, where should we put that?
 So first I want to say is that we have those flops, but they're costing us a lot. And you
 just have to open the papers to see what the cost of the flops is. It's enormous, an enormous energy
 drain. And it's also an enormous architectural drain on what we're doing. And so I would say,
 we want to get rid of the flops, because probably we don't need them. Okay. And especially as you go
 from the data center down to the edge, you get the capability of delivering flops comes directly at
 the, you know, if at the edge, you can put the, sorry, in the data center, you can put, you know,
 your Google data warehouse right next to a waterfall or whatever you want, right, to a source of
 energy, right? When you're doing this on your cell phone or on a tiny device at the edge, every
 little bit of energy that you waste is critical for you. Right. And so what we really want to do
 is move away from the flops and move more towards the very energy efficient way the brains work,
 because this adding more flops is a momentary thing for us. Right. So yes, we can do this,
 but at a very high cost. And no, we don't want to do this forever. We want to find ways to cut the
 cost, reduce the compute. And, and, and there's a little other thing that I want to say, and that is
 architecturally, we generate the flops by running right now, at least by running many, many, many
 tiny cores, thousands of tiny cores, typically, right. And in architecture, in architectures,
 they require a lot of connections to the memory, this high bandwidth memory. And this thing doesn't
 scale. So in a sense, we're trading flops for memory, if you use the CPU today, you could get
 a terabyte on your desktop, but go get a terabyte on a GPU, right. And so using the flops is going
 to enable us changing the architecture, if we don't need so many flops, then we can actually
 increase the size of our memory, which will make us able to hold these giant models that we want to
 do very cheaply, if you will. If I explain a deep neural network to someone, I usually, you know,
 you start with a fully connected layer, you say, you know, here is a layer of neurons, and here is
 a layer of neurons, and they have their connections, right, and each connection has a little weight and
 so on, you usually describe like a dense, fully connected architecture. And that is conceptually,
 I want to say, easy to grasp for people and so on. Do you have an analogy for sparse architectures?
 Like, what is the conceptual like, could you conceptualize to someone who doesn't know what
 like a sparse architecture is and how to think about it? What is different?
 Yeah, the way we do sparsity today, I don't know what it will look like in the future. But today,
 sparsity looks like, imagine that the two layers of the neural network are these kind of, there are
 cords from one layer to the next, right, there are strings attached, and these are, of course,
 these are the connections, the weights that we're using in the computation, right. And sparsity means
 I take scissors, and I chop, chop, chop, chop, chop, you know, till I have five or 10% of those
 cords left, right. And those cords, it turns out, right, if I do this right, if I do this kind of
 pruning right, are good enough to capture, right, the accuracy of the model as it was before, because
 a lot of the connections are not important for this process. That's kind of the big discovery.
 And modern research in techniques for sparsification, right, you know, play along
 this kind of game. So you can do this kind of unstructured thing that I just described, where
 you arbitrarily cut in many places based on the effectiveness, or you can also structurally take
 things out. So in a lot of the modern models, right, we're removing pieces that are not
 necessary. We do architecture search to find these places to cut things, right. So that's where the
 whole game right now of efficiency in neural networks, right, is the game of how do I cut this
 thing down? Right? In the brain, there are certainly some systems like the visual system,
 where that is clearly organized into layers. But there are many other systems that have no
 resemblance to layers, there are connections going up and down and left and right and, you know,
 between the the halves of the brain and all, is there a possible future where this could become
 where this could become into like a standard architectures for neural networks that the notion
 of layers and things like this isn't even really a, you know, a thing anymore? Or is there, you know,
 some some fundamental way where we say, no, there's probably always going to be layers,
 but it's just going to be sparsity between those layers.
 So when we look at, you know, we have a full connectome of essentially only a couple of animals,
 a worm and a fruit fly, that's it. And that's it. You don't see a lot of layering there. It looks
 more like a mess, very sparse mess. Okay. And I would, I wouldn't venture to think about how
 what cortex what a cortex looks like. Right? We don't have that yet. We're working very hard to
 it's a very, these are very hard computational problems to be able to, to go and get a model,
 we just want to do a mouse, even a mouse is just too big for us to do right now, like a small mammal.
 Right. But my, I would venture to guess that yes, the answer is that, you know, it's extremely,
 it's an extremely sparse architecture, and that it wouldn't, it will not look like layers. Okay.
 You can impose a layer structure on any graph. Okay. It's not so the idea that I say there aren't
 layers. Sure. Okay, I can take the graph and I can layer it. Yeah, I could do a BFS on it and layer it.
 But, but the point is not so much that it's more that by design, when I think about it, right,
 I'm not going to think about it as a sequence of layers where the change that I make is the change
 in the layer, one layer is different from the other, but rather, it'll be a combination of
 thinking about paths, different paths, and I'll do different things along different paths.
 That's kind of the idea. You know, if you think about, you know, there's recent research from MIT,
 you know, you can detect, people can detect an image in 0.13, set 0.013 seconds, in 13 milliseconds.
 Okay. In 13 milliseconds, you can detect it, you can say what an image is. Okay. This is,
 there's no time for neurons to fire. This thing is extremely kind of parallel, right, and uses
 very little compute and gets you an answer. And a large part of that is prediction, because you're
 already expecting something. So we need to learn how to do those things. And so machine learning
 right now is in a very naive early stage. And so given that and given the things that we are doing
 right now, it's not, it's not a surprise that we're doing the brute force kind of massive compute
 kind of thing. That's always what you do. And with time, we're going to get better and better at it.
 Right. So that's kind of how I see this progressing.
 Speaking of becoming better, if you know, the flatworm is sparse, the mouse is sparse,
 the human is certainly sparse. Yet our best models today are all big, dense, you know,
 computation hungry things, there is not really a case. Every time I prune, I sparseify and so on,
 I get savings in like, you know, savings in CPU or GPU, I get savings in, you know, my storage,
 but I also get like a little bit worse, right? That's the common thing today in pruning is that
 I get like just a tiny bit worse than the dense model I prune from. Why do you do you think that
 is just the fact that we prune from a dense model? Or what's holding back the sparse models?
 How about if I if I turn this around? Let me turn this around for you. Okay, you can take you can
 take BERT base, which is a common model people use, okay. And you can sparsify BERT base.
 NeuralMagic, we sparsified 95%. So a 95% sparse BERT base, one over 20th of the compute,
 okay, way beyond anything a GPU does, even if you run it with full throttle, okay, it's just
 cutting the compute so much that there's really almost nothing to compute there. It's just moving
 data, okay, not an exaggerating force. But, but you know, it's really becomes a data movement
 problem rather than a compute problem when you when you and you lose 1% of the compute,
 you lose 1% less than 1% accuracy. Okay. And I say, Okay, great. So you've done that, you know,
 and you've gotten all this speed up, but you've lost you say, Oh, near but you lost less than 1%
 accuracy. But what I say instead is forget that. Take BERT large, a much more accurate model,
 several points more accurate than BERT base, okay, and prune it so that it actually, right,
 with 20x less compute, it's actually faster than BERT base. Okay. And so now you have the accuracy,
 right, and you have great compute, and this is through sparsity. So by sparsifying the larger
 model, I actually delivered you the best of both worlds, little compute and great accuracy. And
 that's how I want you to think about sparsity, right. It's a way of enabling us to run much
 larger, more accurate dense models. But because we sparsified them, we are, you know, we're getting
 great performance. That's how to think about. What's the limit currently that keeps us from,
 we always need the dense model first in this model in the pruning in a pruning setup, we first need
 the dense model, then we go to the sparse model, we get huge savings at inference time, what keeps
 us from just building the sparse model in the first place? Great. So this is kind of the lottery
 ticket kind of question, if you will. There is research actually, Dan Alister, one of our
 consultants at neural magic works exactly on this kind of stuff. We know how to run a training
 session right now for four models, where you start out and you need to do only a certain fraction of
 the, you know, of the forward passes, backward passes, dense, and then immediately you can already
 start pruning while training. So there is research going in that direction. But you are right that
 right now at least, right in the standard, if you look at what's going on there out there,
 standardly, you're right. We do most of the time take a standard model and from dense we
 sparsified and so on. But the thing to remember, and this now I'm not talking about the research,
 because the research is going to get there. You know, Janek, I don't know if to what extent we will,
 how fast this will happen and so on, but we will learn how to build sparse architectures,
 it starts sparse and continues, you know, it's really a matter, nature does this. And so there's
 no reason why we wouldn't be able to do it. But I want to say something about today's machine learning
 where you kind of start with the dense and then you have to sparsify. This is really not the
 common paradigm for most users of neural network. For most users, a model is given to them that,
 you know, from a known architecture, right? And then they transfer learn onto it. And most people
 do that rather than train from scratch. They really use the model that somebody already worked very
 hard to build for their specific use case, and then they transfer learn onto it. So this is what
 you can do with sparsity. You can take a sparse model and sparse transfer learn onto it. It's
 extremely efficient because you're running at the speed of the sparse network, right? So you can
 sparse transfer, and then you don't need all of this kind of start with dense. And we're seeing
 more and more sparse networks appear in the literature and in the database collections of
 machine learning models. And as we have more and more of these initial good sparse models,
 right, people are going to learn to start with the sparse already. That's kind of
 commercially, I think that's what we're going to see more and more of.
 Why? You mentioned this a bit already, but why are GPUs so unsuited for sparse models? And what
 makes CPUs in the way you do it, really suited for sparse models? Or are they even suited? Or
 are you simply, you know, seeing that they're better? Yeah, I mean, look, the GPU architecture,
 you know, is designed for this very, you know, small cores, tiny caches. You're not going to go
 and throw all that away just because, you know, you discovered sparsity. So you're trying to
 do sparsity while keeping this kind of lockstep execution structure, right? And this is difficult
 to do sparse. You need really a different kind of setup to get an advantage out of sparsity. Now,
 I'm not, it's not like you can't do that, right? It's not like you can't do that. People can design
 and have designed hardware that utilizes sparsity efficiently, okay? There is such hardware.
 It's just not a, it's not GPU like, it's not like the accelerators that we have today. But all of
 these, again, all of these accelerators have a different problem that has just to do with the
 memory. Because of the way they're designed, right, they typically have very small memories.
 So we're talking, even ones that can run sparse, right, still have the limitation of their memory
 size. So the reason that CPUs are attractive is not so much that, you know, that they, that you
 have a natural way of running sparsity because you can run asynchronous with large cores, but rather
 that the large cores enable you very easy access to very large memory pools, right? So the advantage
 of having strong, powerful cores, right, is really that I can put several terabytes of memory next to
 them, right, and run easily. And that's where the big advantage is going to be. As we understand
 more and more about how to build giant models that don't run all the model layer by layer at the time,
 right, then the compute will be less important. But actually, the ability to hold that model
 in one place and run it rather than break it apart on eight or 16 GPUs, that's going to be your
 advantage. And so this is, so I'm kind of saying it's not so much that you can't build a hard piece
 of hardware to run sparsity, you can, right? But you should build it looking like a CPU in the sense
 of you can access a lot of memory because you're not doing tiny cores. That's kind of, that's my
 two cents. So the CPUs are good because they have, you know, fast connect to large memory, but also
 over the years, we've put more and more levels of cache onto the CPU. How much do you have to
 take this into account when you're building, I mean, maybe you can explain a little bit what
 your company does in terms of software, you build compilers, or can I just run TensorFlow or something?
 Yeah, so let me explain. So first of all, the connection between the CPU and the memory is slow.
 GPU has a faster memory and faster access to it, right? Smaller, but fast, right? CPU memory is slow,
 but large, very large. But CPUs have a cache hierarchy, as you said. And so if you know how
 to utilize your cache hierarchy, then, you know, if you're running in the L1 cache of a CPU, okay,
 you're running as fast as the GPU. There's nothing there that the GPU does that the CPU can't do once
 you're in cache. Okay, in fact, CPU caches are much faster than GPU caches, and the performance is
 better. So the question then, right, and this is what NeuralMagic does is, okay, so what we do is
 we sparsify the model. Now, you know, if machine learning is about, okay, I need to meet a certain
 latency. And because I couldn't meet that latency with a CPU, then we added the GPU and boom, there's
 machine learning with GPUs. Now I can meet the latency. But there's two ways to deal with latency.
 One is to add more flops, and the other is to reduce the flops, right? And so sparsity, instead
 of adding more flops in hardware, reduces the number of flops needed in software. But now that
 you have this very sparse model, because the CPU memory is slow, okay, then what happens is you hit
 a bottleneck, and it's very hard to move. If you do this layer after layer, it's very hard to move
 the data in and out. Okay, so what NeuralMagic invented is a way of running neural networks
 depth-wise. So we have this technology, which we call tensor columns, where essentially you can run,
 okay, you know, you can break the model lengthwise and run, you know, each one of these kind of
 columns, you know, in cache, okay? And because you're not leaving L2 really, or rarely leaving L2,
 you know, you actually get great performance. So in a sense, right, what we're doing is we're
 using the natural ability of CPUs to prefetch things from memory and then run in cache. And
 because this, you know, this cache hierarchy on CPUs has evolved over 70 years, or maybe I'm
 exaggerating, 60 years of hardware design, it's a very, very well understood thing where people
 know how to optimize it, right? Especially the big, you know, chip makers, they really know how to
 make these caches work really well. And so with these really good cache hierarchies,
 you really get great performance by running the model depth-wise. So that's NeuralMagic,
 you know, we take the model, sparsify it, now it doesn't need the compute, and now we run it on the
 CPU and get speed because we're running in cache, okay? And if you look at the numbers, I mean,
 you know, we are, you know, at the speed of, I mean, some numbers we have in publishing,
 we're at the speed of an A100, even faster, in terms of how long it takes. A four-core CPU
 can, in terms of latency, do what a A100 does on a common model at birth, okay? So it's really
 the... Given that it's sparse or... Yes, yes, yes. By sparsifying it and running it,
 you can make a four-core do what A100 does. So it's really now a matter of throughput,
 and the A100 has a lot of throughput, okay? So now the question is, you know, how many cores do you
 want on your CPU to meet the throughput of the A100? And again, the story is that, you know,
 the big providers are adding more and more and more cores, so you're going to be able to
 compete better with the GPUs down the road. So that's kind of the story of NeuralMagic.
 Yeah. So the way I can imagine these tensor columns is that because I execute depthwise,
 the sort of values that I need for the next step in the computation are the results of
 the very last step, therefore, are already going to be in cache. And since everything's sparse,
 I don't need all of the last layer for the current step, and therefore, you know, I have it already.
 Right. And of course, when you think about a neural network, there are overlaps between these
 columns. And the question is, how do you deal with the overlaps in a way that doesn't kill your
 computation? And that's the magic, right? That's the magic of it. There's an algorithm that allows
 you to do that. And because you can do it, you manage to run this way, and you don't hit this
 memory bottleneck, and boom, you're in business. Yeah. So for GPU, it's almost like, you know,
 GPUs enable us to do dense models. But I think also models have almost co-evolved with the GPUs. So
 people have started building models to fit the GPU architectures better, right? Especially
 something like a transformer is like, that's like made for GPUs. Is there a type of model
 a type of sparse model? Like if you if you could wish for the best possible sparse, but you know,
 there's different kinds of sparsity, like, what is the best type of sparsity to let's say execute on
 a CPU? If we want to look forward, and we want to especially build architectures for them?
 Yeah, this goes back to your original, one of the first questions you asked, right? It's about
 it's about a different structure for the neural network execution. So we should forget the
 synchronous layer after layer execution. And think about the fact that, you know, we can run through
 a model, right? In multiple paths with multiple computing units, use the same weight structure,
 and so on of the model, right? But run at different speeds. And by running at different speeds, and
 going through the model in different paths, I can get from the same model, multiple answers to my
 questions, which is kind of what I believe what your brain does. So what happens there is,
 you have this network, but it's not like, you know, it's all firing like this layer after layer,
 it's rather, you have these asynchronous flows going through it, right? Even going through
 matching paths, and CPUs are naturally built for this thing. Now, I'm not saying that somebody
 can't build a beautiful FPGA that will perhaps have a better, closer structure to what a brain does.
 Maybe so, but, you know, but there is an advantage to being commodity. Okay, the fact that the CPU
 can do other things is a big win. If I can move everything to software is really the thing,
 is the thing, then I can really get all the advantages of modern software. So I'm not
 poo-pooing hardware accelerators and saying, great, you know, they have a role and so on and so forth,
 but they come at a price, right? And the price for any organization is that you, instead of just
 downloading or shipping your product with the machine learning piece, you have to ask the client
 to buy a certain accelerator, or run it with a certain accelerator. And this all goes away
 if we can figure out how to make the CPUs do what the GPUs do, right? Then we have, then we're back
 into this beautiful world of containerized, movable software. And that's really kind of where I would
 love machine learning to move to, rather, right? That we would have, and maybe down the road,
 right? There is this, you know, you know, CPUs have a history of absorbing the key components
 of any new paradigm that shows up. You know, virtualization started out with tricks on a
 CPU, and then later on added the features. Networking had special accelerators, and then
 they moved into the CPU. And I'm expecting that whatever features are necessary for machine
 learning to run well, will move into the CPU, and we won't need an outside accelerator to make this
 thing work. If you could. So I think that's, by the way, also the story of GPUs themselves,
 right? They were already kind of consumerish available. And then they can't they absorbed
 machine learning. It's not necessarily the best architecture for machine learning. But
 let's say, let's say there's already all this hardware out there, right? There's very good CPUs
 next to very good GPUs. How do we get the best out of a machine like this? Right now we've advocated
 for let's move things to the CPU, right? We have some advantages there. But what if I have a box
 with both like currently, I just use my CPU to ship data to the GPU, right? That that's what my
 CPU does. But is there a way where I could potentially, you know, what kind of architecture
 would make the best use out of a combined system of CPUs and GPUs? No, I think this is really the
 vision that Nvidia has at least today for their grace Hopper architecture, it's essentially the
 there will be a CPU and a GPU connected to one another. And the CPU will do all the things that
 are memory intense, and the GPU will do all the data intent. The thing about the problem with this
 kind of a model is it's a beautiful model, by the way, I'm not saying anything bad about this. If
 you if you really want to build a GPU world, that's a great thing to do. But again, the, you know,
 how you how much you utilize your GPU, your attached GPU has to do with how you write your
 application, because you need to move the data into the GPU in and out. And that's slow, right?
 You remember, it's like, it's exactly like going to memory, right? It's the GPU is not up, it's not
 sitting in your in your caches. So if you're on the CPU, and you're computing something on a cache,
 and suddenly you get a page fault, and you have to go and get something from memory, that's the
 latency that the GPU introduces you, right. And so if, if you're going to design it with that, you
 have to create really good software to pipeline things. And this is at the level of the application.
 So the application programmer has a big programming task. And so this is a great solution
 for large scale, big projects where, okay, I'm going to Facebook is going to get, you know,
 1000 of these or 10,000 of these, whatever it is, you know, or Google 10,000 100,000 of these and
 put them together with, then it's worthwhile to write this kind of complex software. But if you're
 but if you're Joe company, right, and you have your little thing, I don't think you want to be
 writing that interface, right. So so kind of, so I'm saying it's, it's a it's great for large
 things, right, data center things, big things. But I'm very doubtful if this is going to be
 effective at the edge, if you can actually utilize the CPU for it. Okay. And, and I will say one more
 thing. And that is that, you know, that the modern way that the designers of hardware, think about it
 is that it's mod, it's built in modules, if you look at the, if you look at the AMD latest
 architecture, right, essentially, you have the CC axis. So, so the machine, even though it has,
 you know, maybe 40 or 50 or 60 cores, right, they're grouped into groups of eight, right.
 And each group of eight like this is a little piece of the die. Okay. And I think Intel is
 shifting in that direction, too. So nothing's to prevent you from making pieces of that die
 be specialized pieces of hardware like a GPU, you don't have to have outside device. So if you ask
 me what the future is going to look like, it's probably going to look like, you know, these large
 cores, right, that have, or large machines with multiple dies. And on these dies, we might have a
 GPU die, we might have accelerated. And that's more like what I expect to happen, rather than
 having a massive, you know, accelerator on the side. If we, if we are sparsity, and things not
 being in layers, and so on, naturally, the topic of I think graph neural networks is very close to
 that, at least in the imagination of people, do you have anything to say about, you know, where
 current graph neural networks stand with respect to sparsity?
 Yeah, I would think of graph neural networks as a, as a different kind of, okay, so,
 so graph neural networks, I use some some graph neural networks in my research. And the,
 and the idea there, you know, is that, you know, we can use graph neural networks to solve graph
 problems that otherwise would be very complicated to solve if we tried to solve them brute force.
 Okay, now, it's not generally applicable, there are quite a few limitations. But,
 but as a tool, I would say that, you know, rather than think about the neural network itself as being
 looking like a graph neural network, right, I could use graph neural networks, right, to define
 what we call motifs in the neural network. So for example, when we try to look at,
 at how brains are structured, right, when we look at the graphs of brains, and we try to understand,
 you know, is there a motif that is repeating itself in this graph, right, then using a graph
 neural network for that is a really nice way to try to find these motifs, okay, efficiently, right,
 because the problem itself is piece-based complete, or we don't know, it's graph isomorphism. So,
 so clearly, we don't know, right, how to do the brute force algorithm well. But,
 but the graph neural network can come to our aid here. And so, so I would say that right now,
 I don't really see a real network design, neural network design that is specific to that,
 or a way that it helps. But, but in research, it definitely helps. And we really want to use these
 networks to help us in research. This might be a bit of a tech bro question. But if I hear,
 you know, I can do sparse computation, very, I can reduce the flops and so on. Is there
 any intrinsic connection between the sparsification of neural networks, the non layer
 wise computation, and blockchain technology and smart contracts and distributed computing and
 things like this? Have you ever given this any thought or or? Yeah, is that completely off?
 Yeah, look, I think nothing is completely off with respect to machine. That in the sense that I am
 sure that machine learning will find its way into into all of those areas, right, it's a matter of
 time. And, and right now, right, the all the work there doesn't need the efficiency of, of, right,
 of what machine learning offers, because machine learning, in the end, is an optimization technique.
 And so when I think when all these blockchain algorithms and all, you know, become more common
 place, and we need to provide them with things like security, further security or analysis,
 and so on, I think then we're going to see applications of machine learning there. And with
 that, I think all these things of sparsity and so on, I think are going to appear. But, you know,
 but but for me, right, it really is the whole story of sparsity, right, is the story of a
 of a phenomenon that is very prevalent in nature, right, that may you can say, surprisingly or not
 surprisingly shows up in machine learning. And it kind of it makes me feel like it's strengthening
 my belief, right, that even though the exact computations that we're doing are not the same
 as spiking neural networks in brains, right, that there is a lot of commonality there.
 And the emergence of these similar phenomena, like sparsity, like, you know, pruning and so on,
 and the fact that we can get benefits from it, this tells me, oh, okay, these are related.
 I think that's a very important point to keep in mind.
 With neural magic, who is your main target audience? Like who who is listening to this?
 Do you want to let know like we are exactly for you?
 So we span the gamut from the data center to the edge. I would like to say, I mean,
 we just now are moving into providing the same properties for ARM architectures. And so I would
 say the exciting new thing in neural magic is we're moving from doing this, you know, for AMD and
 Intel architectures to doing it for ARM, which means that we're going to span the gamut all the
 way to the very bottom of the of the food chain, if you will. And I think this is very exciting,
 because as you know, because because sparsity has a dual role as you go down the food chain,
 right, because for the large accelerating, you know, the fact that the memory footprint is large
 is small is not that important. But as I go down, sparsity gives me two things speed with neural
 magic gives you speed, but it also makes the model extremely small. So you're getting a small,
 accurate model by running on a very small device. And this, you know, typically is an ARM device.
 And so that's, that's, that's the audience that I'd like to say, hey, we're coming, you know,
 we're coming in, we're going to deliver the same things that we can deliver for Intel and AMD,
 we're now going to deliver it for ARM at the very end.
 If you say edge, do you mean smartphones? Do you mean security cameras? Do you mean robots?
 Everything? Okay. I mean, everything. I'm not like I'm going to do everything to start with. But yes,
 yes, we're aiming in that direction. Yes. And with the danger that this is become going to become
 like a marketing opportunity question, but how easy is it to get started with what you're doing?
 Like, let's say I'm, I'm like, I've done, you know, my TensorFlow tutorials, I know how to build a
 model and train it and so on. Like, how much does it take for me to transition or to apply what
 you're doing? Yeah, so you just go to our website, go to get go to get download deep sparse, our,
 you know, our engine download our ML tooling. And, you know, immediately, you just either pick a
 sparse model and transfer learn onto it with our tool. So we have recipes, you have a model,
 you have a recipe, exactly what you would do if you went to hugging face and downloaded a model and
 download a recipe, you do the same kind of thing. And you sparse transfer learn onto it,
 and you're in business. So it's not very hard. So I think this is really and we're working on making
 it even even easier. This is one of our goals, right is to make it really, really easy to do this.
 And the advantage of course, is that, you know, people are already busy, you know, quantizing
 their models to get more performance. So this is like quantized, in some sense, right, you're going
 to do the same kind of thing and get a lot more performance. Is there a type of model where it
 works particularly well and the type of model where it doesn't like I'm thinking, you know,
 conv nets, recursive networks, autoregressive, maybe, you know, the big language models,
 like what what is it best at? Yeah, so right now, you know, it's best at at bird YOLO models,
 we do we do computer vision, and we do, and we do the language models, but not the large language
 models, we haven't done the large language models yet. So for those types of things like the birds
 and the YOLOs and the, you know, the whatever the variants of efficient nets and all these guys,
 this is, you know, visual transformers, these are the things that that we do right now. And
 and all our technology is right now, you know, available for those, I'd love to do the large
 models, a CPU is a natural environment for running the knowledge models, you know, these giant models,
 these trillion or whatever parameter models that people talk about splitting across 16 GPUs,
 they fit on your desktop. Okay, so clearly, a CPU is a natural place to run a very large model. Okay,
 and so that's that will be a target, but rotten, but not right now. Okay, very exciting. Is there
 any last things you want to get out maybe about neural magic or sparsity in general? Well, you
 know, our our whole machine learning software stack is open source. And we'd love people to
 come in and help us build, you know, better sparsity use sparsity in their models and,
 and tell us about what they're doing. And, you know, that it would we have a community,
 and we'd love you to join us. Excellent. Nier, thank you so much for being here today.
 This was very pleasant. Thank you very much. Bye bye. Bye bye.