Sparse Expert Models (Switch Transformers, GLAM, and more... w/ the Authors)
#nlp #sparsity #transformers

This video is an interview with Barret Zoph and William Fedus of Google Brain about Sparse Expert Models.
Sparse Expert models have been hugely successful at distributing parts of models, mostly Transformers, across large array of machines and use a routing function to effectively route signals between them. This means that even though these models have a huge number of parameters, the computational load for a given signal does not increase because the model is only sparsely activated. Sparse expert models, such as Switch Transformers and GLAM can scale up to trillions of parameters and bring a number of desirable properties. We discuss everything from the fundamentals, history, strengths and weaknesses, up to the current state of the art of these models.

OUTLINE:
0:00 - Intro
0:30 - What are sparse expert models?
4:25 - Start of Interview
5:55 - What do you mean by sparse experts?
8:10 - How does routing work in these models?
12:10 - What is the history of sparse experts?
14:45 - What does an individual expert learn?
19:25 - When are these models appropriate?
22:30 - How comparable are sparse to dense models?
26:30 - How does the pathways system connect to this?
28:45 - What improvements did GLAM make?
31:30 - The "designing sparse experts" paper
37:45 - Can experts be frozen during training?
41:20 - Can the routing function be improved?
47:15 - Can experts be distributed beyond data centers?
50:20 - Are there sparse experts for other domains than NLP?
52:15 - Are sparse and dense models in competition?
53:35 - Where do we go from here?
56:30 - How can people get started with this?

Papers:
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (https://arxiv.org/abs/2101.03961)
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (https://arxiv.org/abs/2112.06905)
Designing Effective Sparse Expert Models (https://arxiv.org/abs/2202.08906)

Links:
Merch: http://store.ykilcher.com
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hello, today I'm having an interview about the topic of sparse experts. Now, ironically, the people are absolute experts in this type of models. These models, they are huge, they're usually language models, but they don't have to be they're usually transformers, but they don't have to be what they do have in common is this notion of sparse experts, these models go up to the trillions of parameters, and they achieve this via sparsity. Now I want to do a very, very brief introduction of what sparse expert models are. And then we'll dive into the interview right away because I don't want to keep it from you. So let's look at a transformer model. Usually, I have some sort of an input that is tokens, a sequence of tokens, which are represented here by circles. And what I'm going to do with these tokens is I'm going to alternatingly push them through different layers. Now one big layer type that is common in transformers is the attention layer, we're not going to talk about the attention layer today, all you have to know is that it takes in a sequence of tokens, and it outputs a sequence of tokens again, ideally the same amount as went in, which I failed to draw here, the other very common big type of layer in these transformers is what's called the feed forward layer. Now the feed forward layer is just a linear layer, and every token goes through this linear layer by itself. So every token individually goes through the same transformation. And thus, as we do this with all tokens, again, we end up with a sequence of as many tokens as we input. Now a sparse expert model isn't very different than this, the attention layers commonly aren't really touched. So that works just the same. However, in the feed forward layer, we see a big difference. Notably, we don't only have one feed forward layer, we have many. So here is feed forward one, here is feed forward two, here is feed forward three, and here is feed forward four, each one representing a different individual linear transformation of a token. Now when we talk about sparse experts, these things here are called the experts, they're called the experts because they're thought to specialize in very specific tasks. And the goal in sparse expert models is to route the tokens to the corresponding correct experts. So every token goes through what's known as a routing function. We're going to talk about this routing function in the interview. But in essence, it is a very simple, usually something like a linear function or a simple transformation that decides to which of the experts any given token is routed. So sometimes even in sparse expert models, a token is routed to multiple experts. But in the newest iterations, the tokens are simply routed to one single experts and none of the other. Usually this is done, as I said, by some sort of a linear transformation, followed by a softmax to decide where the token goes. So every token would be assigned to one expert. And that gives the possibility of scaling these models up dramatically. Not only do you save a lot of compute because the tokens only go to one place ergo, you only need to compute that one thing for that particular token. But also there's the opportunity to massively shard and parallelize these different experts across different machines, as you only need to route the token to one place. That means you dramatically reduce these big all to all reductions, they still happen, but not as much. So as I already said, the biggest models have trillions of parameters, you need to take a little bit of care of how you then aggregate the tokens once they come out of the experts. So essentially what you want to do is you want to carry over the likelihood from the routing function up here. But this is a minor detail, a minor details are important, but you know, so I know it doesn't look like much, but these sparse expert models really have the potential to massively scale up our current efforts in AI. And I have no doubt that they're going to play a role in the near future, when we're looking at bigger and bigger models, because at some point, the purely dense models will reach sort of the limit of what's physically doable. And then it's a good opportunity that we have models that can go even larger. Alright, so without further ado, let's jump into the interview. I hope you're enjoying yourself. If you do have any sort of comments, please leave a comment, share the video around if you like it, and I'll see you around. Bye bye. Hello, everyone, my guests today are William Fedes and Barrett Zoff, who are engineers and researchers at Google, Google Brain, and have been diving into large models, specifically sparse expert models, which are models that, well, feature this notion of experts, and also have a notion of sparsity. And hopefully today, we'll discover what this is all about. Specifically, we'll talk broadly about three papers in a long line of work. One is the switch transformers paper, which was really, I believe, one of the first papers that just had like massive amounts of parameter was that like trillion, probably trillion parameters. It was big. 1.6 trillion parameters. That's right. Yeah, yeah, it's insane. And then there's there's glam, which demonstrated really nice scaling laws with these sparse experts. And more recently, there is designing effective sparse expert models, which as far as I can see, is also a bit of a of a maybe a summary recommendations, more of a what we learned type of thing. So William and Barrett, welcome to the channel. Thanks so much for being here. Yeah, thanks for having me. So can you give us just a little bit of context what you mean when you say sparse expert models? Yeah, sure. So this is a great question, especially since the word sparsity crops up in like many different aspects of deep learning, whether it's, you know, like sparse attention or, you know, various other sparse paradigms. So yes, sparsity in our case means that each input can get different subsets of parameters. So that's kind of like the main sparsity that we're talking about here. And it's like, you know, it's a very natural concept, right? Like normally, in like a dense transformer, for example, you have, you know, a word embedding, and, you know, any word will have the same parameters and compute applied to it. And in sparse models, typically what happens is you have the same amount of compute, but you can have different subsets of the model parameters be like, you know, acting on the model inputs. And what does that mean in in practice? So we're talking mainly about, let's say transformer models here. No, is that a good characterization of things? Or do you do you see sparse expert models in a more general sense? Yeah, I mean, these things actually almost sort of like cropped up originally as almost like in the context of like ensemble type methods, where you have a bunch of like almost like fully independent models. And then you're sort of using these as like, you know, each model as an expert. But the common paradigm as of like 2022, is sort of experts as a layer. So this is like really popularized by Noam Shazir's work in 2017, outrageously large models. And in that context, they were actually inserting it in between LSTM layers, which is like the prevailing like recurrent architecture at the time. Most of the things just because like the world has sort of shifted towards transformers in it seems like almost all modalities now, we're often thinking about experts as a layer inside transformers. Typically, we're sort of doing this at the feed forward. So these blocks that just sort of independently apply on the different like tokens. But we've also kind of considered it in self attention layers, it's just sort of like a very general concept. But yeah, typically in transformers. So you have this notion of an expert, which you say is is sort of a specialized function or something like this. And then there's often this thing called a router. How how does information find its way through these experts? What are the general principles in that? And why would I even consider doing something like this? Yeah, so great question. So yeah, so you have this figure up here. And so one thing to notice that basically, if you only have a single expert, it essentially reduces to just a normal dense transformer. So the interpretation is pretty natural. And in almost all of the ways people are doing sparse expert model nowadays, there's some notion of a learned mechanism that for, you know, embedding at the current layer, you figure out what expert you should send this representation to. And this can be ranging from very simple to just like a simple softmax function over the total number of experts to very complicated linear programming type solutions that have a more like globally optimal solution. So yeah, so this is this is kind of like the paradigm. And I think it's a pretty natural one. So even if you want to only, you know, yeah, apply one set of weights per representation, now you have the option of just instead of always applying the same weight matrix. Now you can, you know, maybe have a selection of in this figure for different weight matrices. And the way that, you know, we've done this in our work, and I think is the most common is just as a single feed forward network. So you take your input representation, and then you just, you know, apply it with something that's going to be like, you know, the model dimension by the number of experts, and then you apply like a softmax function to get like a probability over all of the different experts. And our switch transformer work, the routing was extremely simple, where it's just like you just send it to the highest, like the highest expert with the highest probability. And then, you know, you just simply route it to that expert, then the output of that computation gets scaled by the router probability. So if it was like, oh, with 0.9, send it to expert two, then when you have the output of that computation, you scale it all by 0.9. Do I remember correctly that there was some paper that it was this an older paper, and this might be getting very technical for a second, but was there an older paper that said something like you always needed to send it to at least two of these experts, otherwise, it's kind of unstable. Is that an older paper, or a newer than yours? It actually wasn't instability that they're clashing against. It was more this idea that we're doing this like weird discretize operation. So instead of using like reinforcement learning to sort of like update on the experts, we're kind of doing this like kind of hacky back propagation through these like softmax operations, which have been masked. And the idea that top two or greater was necessary because they were thinking, well, I'm creating a probability distribution for this token for this word over the available experts. If I don't have at least two, I can't tell whether expert i or j was sort of better for this one. So it's like, in order to have the hypothesis was sort of like a useful gradient signal for the router, it has to know, well, should I have sent it to i or j? And then we just sort of didn't follow convention and did one. And it also seems to work just fine. I think in part because you're sort of doing this sort of normalization. So you can still get an up waiting or down waiting if you select an expert. So it's like, oh, if that expert selection worked out well for you, or worked out poorly for you, you can then sort of adjust the embedding for that expert. And then you at the next pass, if you saw that same token, you're still doing this like softmax distribution. So you're kind of like up waiting or down waiting it. So I think that's sort of like the gist of the mechanism. And this, this, I think this idea was at least from 2017, it may have predated it. Could you maybe now that we're talking about history, trace the evolution of this line of research a little bit. You already mentioned this existed as sort of ensemble methods inside of it. I'm talking now specifically about sparse experts within transformers, which are the things that allow us to really scale up to these giant models. What's the what's sort of the line of research? What are the original things? I'm going to guess this this work is among them. And what were the improvements that happened since then in this field? Bear, do you want me to go or you go for it? Yeah, so I mean, like, going back 30 years, like you have like Jordans and Jacob, this obviously predates transformer because transformer was a 2017 development. So I mean, the concept is very, very old. I think it just kind of like resurged in popularity. I'd say the first, yeah, the very first sort of use of mixture of experts in transformer was left in it all in 2020. So this is G shard. And it just showed really remarkable improvements in translation. What they were doing was, you know, analogous to switch transforming these other works is they just sort of substitute these feed forward blocks with experts. And in that case, sort of also similar with switch transformer, they had many, many experts, I think in that case, it was thousands. And they were showing really significant improvements over state of the art translation models. I think as the field has sort of evolved, as we've sort of like learned a bit more about it, there seemed to be this like kind of general trend of like, okay, cool, we can pre train these models or like in the case of translation, there's no big distribution shift. When you're training to translate, you're also doing inference to translate. But in switch transformer, we found, okay, we'll pre train to, you know, improve the perplexity, improve the prediction of next token. And we were getting significant improvements. But then when we took it under a data distribution shift to fine tuning, it was performing quite badly with many experts. So I think there's been this trend to try to balance the computation and the parameters a bit more. So I think some of the prevailing models have actually in transformers have actually gone have actually gone towards fewer experts. So 1632, 64 experts, not 1000s of experts. So that's kind of like the lineage of mixture of experts and then like mixture of experts in the context of transformers. And what is so in that context, if one expert is the classic transformer model, and that seems to not work as well as many experts, but too many don't work, what is the abstraction that I can think of for an expert? Like, what does an expert learn? What is an expert responsible for? Approximately? Do you have any idea what happens? Like what, what, how does it make sense that the optimal number is, let's say, a few dozen and not super many, but also not one? Yeah, so great question. So yeah, there's like a few parts to this. So one, like, I think it's really just like an empirical observation right now that, you know, 16 versus 64 versus, you know, 2048 versus 10,000. You know, like, it seems like the expert numbers in the middle, like, it's not from the standpoint of like on a per step basis, more experts typically don't make things worse. Usually it's like better or about the same, but things start to level off. But it's very inconvenient to have a lot of experts because it's just this like a huge memory footprint, the way that the models are distributed, it's not really amenable towards typically, unless you have like tons of, you know, parallel cores going. So like actually the observation where you kind of want to actually have like a middle amount of experts is a lot of the times actually driven by just the like practicality of then like training, serving these models. Yeah, in terms of like, what these models are actually learning, like intuitively. So we actually studied this in our most recent work, kind of looking at, you know, each expert, what are they specializing in, what are they learning? And interestingly, they kind of specialize in some shallow concepts, which you would think maybe there would be like only really deep things going on. And it would be kind of hard to inspect them. But you know, we noticed like, oh, there's like a punctuation expert, or an expert that will, you know, talk about, you know, like proper nouns, which we thought was pretty funny, and maybe not super intuitive for, you know, how. Yeah, actually, if you want, you can switch over to the recent paper, and we actually have a figure which sort of shows some of these things. So you can kind of like follow along and see how shallow these things actually are. Yeah. Yeah. So this this would be this would be different. So you you found an expert or in this case, multiple experts that that focused on the these sort of things. So there's conjunctions, punctuation, verb, visual description, which is which is interesting, because that's kind of I want to say like a higher level thing than just the punctuation, right? Counting numbers. Yeah, how do you make sense of this stuff? Like, what's going on? I Yeah, I mean, I think we were sort of expecting maybe like a higher level of description, but like, or like, sort of like representation. Um, it's, I think we've just started started to sort of like crack and like, look into these models to actually see what's going on. That obviously, like one big specialization that you're seeing here are these Sentinel tokens. To make sense of that, we were sort of doing pre training where it's sort of fill in the blank test. And a blank is sort of represented by these like little Sentinels. So like extra ID 10 represents like, you know, the blank 10. And we often really frequently see experts are specializing on these blanks. So, like, we're doing pre training. So that's sort of an interesting thing. And then I think that also might segue into maybe you want to actually, given this sort of like, you know, observed specialization, maybe you actually want to make some experts higher capacity or give them more compute to sort of do things that might be harder. But honestly, I mean, this is still very early. It'd be interesting for sort of like, you know, some of the interpretability lens that like entropic has on some of the recent sparse expert models. Some questions we've kind of received are, what is the interplay of expert specialization with sort of like self attention specialization? And that's honestly completely open. I think we were just sort of putting this table forth to the community to be like, well, we, we started, it's not exactly what we would have expected. But definitely kind of like a call to dig further and hopefully like, you know, further improve things with the also I believe that this was Oh, yeah, here already in switch transformers, this ability to distribute these things across devices that comes naturally with, with having sparse experts. So sparsity meaning in this case, I only send stuff to one or a few experts. And there there came the ability to shard this across devices, how, like, how practical is this really to like, what, when would I do something like this? At what point would it become practical and useful and the best thing to do to communicate across devices for my experts? Yeah, so really great question. And I actually think this is the reason why the method works so well, actually. So the standard way I would say people are doing distributed training of these models is they have, you know, either fully data parallelism, which means like, you know, each machine has the same set of weights, but different slices of data, or a blend of data and model parallelism, where it's like, you know, kind of a mix where certain like, you know, cores have sometimes different weights or sometimes different data, and then you communicate stuff to make it, you know, emulate like a full model. But I think experts, one really easy interpretation of this is like, let's say you have a model, and, you know, you're using data parallelism, and you have four different machines, a really natural way to overlay experts on this would be you just have one expert per machine. And then, yeah, so this is like a really nice interpretation, because then when you have all of your, you know, local data per core, you'd have the router weights replicated, but then you just figure out what expert they need to go to. And then that's when you kind of, you know, shuffle all the tokens around to the machines, do all the computation, and then shuffle them back. And this makes it really nice, because then per machine, you actually never have any more parameters than you would have had just with the Dense Transformer. But now you have experts. So it's actually like a really nice way of kind of, you know, thinking about how to design the models would be like, oh, you know, you have this many cores for data parallelism, just have that many experts. And that's actually a paradigm that Bim and I use a lot when designing these models as well. And yeah, I mean, I think as soon as you have this sort of like, distributed model, where you're already going across accelerators and devices, you do already have these communication patterns, right? Like you need to get activations to a certain place, you need to like get gradients to a certain place. So you already have these sort of like all reduced communication collectives. Expert model is going to introduce all to all communication patterns. So that can be like a more expensive thing, especially based on like your topology and the bandwidth between all of your networks, or between all of your devices. But yeah, so I mean, this is something you sort of have to like, kind of empirically test like, okay, how much does this architecture kind of buy you in terms of performance on your task, versus the additional costs of all to all communication. But you will be communicating across devices for these big models, regardless to train them. Yeah. So this is a good, I guess, a good segue, because you can achieve these giant models, like trillions of parameters using these is the sparse expert models, because naturally, I can parallelize these experts, it doesn't cost me really much more compute, because any data point, or any token only goes to one single expert. There is always a bit of the, let's say, the question of how comparable this is to the dense models. It was it was often I don't know if this is a latent feeling that I get from the community, but people would rather have the 175 billion GPT three model compared to the switch transformer, even if it is trillions of parameters. Is there some sort of division factor where I could compare to a dense model? Or do you think that it's an entirely different nature of function that's computed here? Yeah, so this is a really great question. And I think there's a lot of different ways you have to kind of look at this to figure out if a sparse model is right for you. So I think actually, in a lot of applications, if it's like, hey, I want to train the model with the smallest memory footprint, so I can just be using it on the smallest amount of devices as possible, a dense model will always be better. Like I think on a per parameter basis, dense models are going to be performing better. So for those types of applications, I'm like, yeah, I don't think it makes sense to be using sparse models. Maybe you want to just train the best thing that you can fit onto your local 2 GPU machine or like a 10 GPU machine, and do really kind of low throughput, feeding in data to this, like not high or anything like that. I think sparse models are good, where you're going to be training a model and you're going to be hosting it on a lot of machines and you're going to be having a lot of high throughput going through it. So a lot of queries, a lot of stuff going through it, because then things can be batched together and then the models actually become pretty efficient. So I think that's kind of one lens to look at when you would want to use a sparse versus dense model. And I think the kind of second lens is that, for a given amount of GPU or TPU hours on a compute cluster, what model will get you the best performance? And I think that's the lens that we actually would spend a lot of time looking at for pre-training models in this paper, like, oh, you have 512 TPU chips, and I give you X budget training hours, is a dense model or sparse model going to give you the best pre-training performance? And I think our assessment was that, yeah, I think actually the Pareto optimal model typically is a sparse model in that setup. Yeah, and comparing parameters, especially between a dense and a sparse model, is just totally incomparable. So using GPT-3 and then our largest switch transformer model, it's just wildly different amount of computes in our case. You can't infer that from the parameter budget. So I don't know what the compute ratio was between the two, but far different. Our 1.6 trillion parameter model was actually only doing about as much compute as a billion parameter model. So for each token, it was doing roughly a billion parameters worth of flops. And whereas GPT-3 is doing 175 billion parameters worth of flops. So you can sort of tune this, and DeepMind has sort of also tried to come up with a characterization of scaling properties, far more robust than we've been able to do, of sparse expert models, and try to come up with a dense model equivalent. So that might be an interesting work to refer to in the future. But really, it's just like, practically speaking, it's like, OK, I give you these accelerators for this amount of time. What's the best model? So that's probably the fairest comparison. Have you seen this Pathways paper? Yes, definitely. They came out. How does it play into something like this? Is it going to make this easier? Is it going to make it superfluous? How does the ability to schedule things heterogeneously across devices, or does it enable new possibilities in the sparse expert world? Yeah, so great question. So one thing to note is, OK, so typically you have dense models. And a dense model, like every input, will have the same amount of compute and parameters applied to it. And sparse models, now you have the same amount of compute, but different parameters. And I think the kind of natural next step that I think makes a lot of sense to both Liam and I is that now for each input, you have a different amount of compute applied as well. And I think Pathways is really exciting, again, like you kind of mentioned for the heterogeneous compute, where we want to have inputs that might require different parameters and also different amounts of compute. Yeah, and I think a framework like this is going to really open up a lot of really exciting research avenues along that direction. And I think it feels like a very natural interpretation for kind of where our models are headed for in the future. Yeah, like right now, it's like our experts are all sort of completely homogenous. They're all the same size. They do the same operations. Pathways, you could be like, oh, this is like a recurrent expert. This is a huge expert. There's a group of small experts. You could just be a lot more flexible in design. And sort of like alluding to that a little bit with when we were sort of looking at the visualization, it's like, oh, wow, a really consistent thing. Our experts that want to specialize in these like fill in the blank tokens, these Sentinel tokens, perhaps that might be an avenue or an area where it's like, oh, let's dramatically increase the compute here. This is, oh, hi, Kat. This is like an area where we like a lot of extra compute could really be helpful. And there wasn't really an effective way to do this with the existing infrastructures before pathways. Is there a... Yeah, sorry, that's lost the train of thought. Explain to me a little bit how GLAM improved upon switch transformers. Like what's new? What's exciting there? Yeah, so I think GLAM... So one also thing to note is like there's kind of a right now division of two different types of model classes in language modeling space, I would say. So one is like these decoder only models where it's just a single set of parameters and it's like you're just predicting the next token like autoregressively. And this is like what GPT-3 is. And this is also the kind of architecture that GLAM studies these models in. So the other classes, these encoder decoder models like T5, this was also G-shard. This is kind of what also we studied in switch transformer in our most recent work as well. So I think GLAM did a few things. So one, they really, I think, pushed the scale of these models. So like while our original model of switch transformer had more parameters, like GLAM had like much more compute applied per token. And they studied these very extensively with decoder only language models. And yeah, I think their main comparison point was to GPT-3 as well. So they were studying a lot in the context of few-shot and like one-shot evaluations, whereas I think a lot of our work actually centered around like fine tuning the models. But yeah, I think GLAM really like pushed the scale of these, especially in these decoder only language models and showed that like, yeah, you know, you can get as good of quality as GPT-3 with like, you know, huge computational training savings as well. And they did a really, a lot of really good work in that space. Is there a functional difference between the sparse expert routing or anything around this in GLAM? Or is it mainly what you said with decoder only and applying more compute scaling it up? So actually, there is a few differences that are more nuanced and technical. But yeah, at a high level, you know, there's a routing function, and they actually route each token to two experts. And actually, there's like some of the differences in these models comes from like how much buffer you give each token, each expert, because, you know, you need to have like fixed batch sizes for all the experts ahead of time. And so what can happen is like, you can't guarantee that like, there's going to be perfect balancing among all of the tokens getting sent to experts. So like experts can overflow. And there's this key parameter that we call the capacity factor. That's probably the single-handedly most important parameter when designing a mixture of expert models, because it just has such a huge impact on the communication costs, compute and everything like that for how much buffer you should have. And yeah, I think a big difference from GLAM versus our models is they actually use like a much larger capacity factor than we've used in our other works. But yeah, the routing algorithm is essentially the same. That is, yeah, I want to get a bit more into the into the routing algorithm in just a bit, but just to end this with the with the last paper that we've previously looked at, was I right in saying that this is much more often, let's say a general, almost like a review paper? Or how would you describe it? Yeah, I mean, I think we we tried to make sure like we're contextualizing a lot of the work. So we tried to make sure the related work was like, pretty inclusive, because I mean, I think the field's really adjusted and improved a lot in the last two years. But I would sort of characterize this paper as fixing the two big flaws from our first one from switch transformers. The first was these models are unstable to train. So we'd be training and then all of a sudden the loss or just diverge, which thwarted a lot of our issues. Interestingly, it doesn't seem like the instability arises from a lot of experts. We were consistently able to train models like our trillion parameter model, for instance, with thousands of experts, never really hitting any unstable sections, really kind of came from like high clops or high computation expert models, even with like few experts, those were highly unstable. And then the second thing that this paper sort of fixed was the sort of like poor fine tuning quality. So we would sort of pre train a model, it would show like really significant speed ups over a dense counterpart. But then when it came time to fine tuning, say I'm like super glue or some like other task of interest, it would just be considerably worse. So I think this paper was just really trying to sort of like kind of patch up a couple of those issues, we identified them in our first work. Yeah, I'm always a bit intimidated when a paper has a table of index by itself. Can you can you go to something that Barry and I discussed, it's like, okay, should we break this up into multiple papers? Or should this be one because, you know, this is like, you know, a lot of work. And, you know, this is like something that we discussed, like maybe in the future, we should probably be producing like more bite size pieces of work. When you when you talk about fine tuning, can you go a bit into more detail? Like, what was exactly the problem? How did you how did you also go about fixing it? So I'm not only interested in, you know, how did how what's the final model like, but what does the process of debugging something like this and then getting to an architecture or a solution that actually works look like? Yeah, I mean, it's sort of this like very interesting problem of like, you want to, there's really just like fundamental trade off. And whenever you're sort of doing a sort of like large scale work, where you want to try to understand and characterize things at a smaller scale, understand scaling properties, understand, understand like hyper parameter dependencies. But then you also want to be consistently checking yourself at the largest scales. And this sort of balance of like, okay, you have this much compute, you have this much time, where do you allocate it? Do you do a lot of small experiments? Or do you do a few big experiments? It's kind of tricky. But I'd say part of our like findings were the first one was like, okay, well, characterization is we're not doing better on fine tuning. What's the cause? And it seemed like perhaps our cause is not that of optimization, it's that of generalization. So if you scroll down into section four, you can just click on the link. We might be Yeah, exactly. Yeah, so this is an example that, you know, kind of supports a lot of the trends we're seeing. On the left is a small superglue task. So this task has only 250 training sequences, so very small. And on the right is record. So this has over 100,000 training examples. We're showing sparse models versus dense models in the two things, in the two plots. Blue represents the sparse training though, and you can see it just very quickly gets to 100%. And it outpaces in both cases, the small task and the large task outpaces the dense model getting to 100% train evaluation accuracy. But when in the small task, we'll see the dense model in red actually outperforming the ultimate performance for the sparse model in orange, whereas for the bigger tasks, the sparse model does well. And so we kind of kept seeing this like, you know, overfitting issues. And a lot of this was then led us to sort of like investigate hyperparameters. And, you know, some of the hyperparameters can sort of be adjusted in a way to make the model like less susceptible to overfitting. So you can use like different dropout parameterizations, but also things like batch size and learning rate can inject more noise, which can also be sort of like a counter to some like overfitting properties. So we tried and then sort of consistent with this, like a lot of these things were sort of like, you know, more exhaustive studies at say, a billion parameter scale, we then tried to continue to sort of like fact check this against our larger model, and make sure that these conclusions were holding. So I think it was just sort of like, you know, the debugging process was, okay, what more precisely is going wrong? And then like, what are our levers that we can sort of like pull in order to try to like improve it? But you know, a bit of art and science really. You so you is it you observed, okay, we are probably overfitting, because you saw the smaller the tasks got sort of the worst the sparse models would ultimately perform on the validation set of those tasks. Did you? And you have it's not like quite like, yeah, it's not always like quite so easy as that. But it's sort of like, you know, directionally, like, I think we have support of the hypothesis. But it's not like every single small task does poorly. And every large task is great. Yeah, but I'd say directionally, it seems to be a phenomenon we've observed. You have also a bunch of experiments down here where you where you investigate some of these, for example, dropout probabilities, you also have expert dropout probability, which is one of the questions I had in that you have particular architecture, right with these with these experts. And when I think about overfitting, what in regular transformers, I have kind of handles, I can use adapter layers, I can only fine tune the head and so on. Did you ever investigate maybe only fine tuning some of the experts? Like is that the keeping others constant? Is that ever a thing? Like, would that work? Or, or, or, you know, can we can we make use somehow of the fact that we have these different experts, and they're actually different functions? Yeah, great question. And I think actually, if you scroll down, we did a very naive kind of version of this, not where we freeze different experts, but we, you know, freeze all of the experts, or maybe only train all the experts and freeze all of the other parameters. I would say our findings were this were surprising in a bad way. So nothing, nothing really worked super well. So here you can see that and this is also, we only studied this on super glue, right? So it's far from exhaustive. But yeah, so one thing we tried was updating first all of the non mixture of expert parameters only. And that actually performed about the same, which was kind of interesting. It's like, hey, like actually freezing the mixture of expert weights like seem to perform about as well as just like updating the whole model. Then when we started to, you know, update only the mixture of expert weights and freeze all the other model parameters, like the performance was actually really bad. And there was some we still fully don't understand what's going on here. We have like a few kind of like half baked hypotheses. But yeah, then when we update only the attention parameters, things are worse. And we found a slight boost updating only the feed forward network parameters that weren't the mixture of expert layers. But yeah, overall, nothing worked that well. But yeah, I think there might be some potential really interesting things of like, hey, maybe allowing only, you know, a certain subset of experts to be fine tuned. We did spend a little bit of time actually studying like pruning off experts during fine tuning. So like for a specific fine tuning task, if your pre trained model has like 64 experts, can you just take like a subset of like two, four, eight or 16 of them? Yeah, and we also didn't really get that good of signal with this as well. Also to some of your recommendations, they actually would be compatible with expert models too. So you're free to just like fine tune like the top, like top logit layer, or you could add in adapter layers. Yeah, we didn't do anything like really funky, like you were suggesting like, oh, we're only going to expert like update experts like three, eight and 14 or something. Yeah, my intuition is that probably wouldn't work well. But I mean, I've been proven wrong many times. Yeah, we tried some like other things that didn't make it to this table or these plots. And yeah, again, we didn't really see like a significant boost. That said, if you are only updating like a fraction of the parameters, you get some memory savings. So you know, some nice things. Cool. I guess one, you know, there's, there's almost an infinite number of things one could try with these things like distilling experts like distilling multiple experts into a single expert. So you have another expert that's again free to do some some new tasks. Once you know that two experts are converging something like, I think there's, it's really interesting, right? A lot of we're adding a new experts on the fly. Yeah, a lot of possibilities. And that brings me a bit to this this routing function that we talked about before and at the beginning, which seems to me is a really crucial part of the system. Yet, as you said before, very often, I've just seen this being implemented quite simplistically, maybe there's a linear transform and then a softmax or something like this, maybe not even maybe there is some some sort of a, you know, a some fixed keys for all of the experts and then you route according to that. Do you like my intuition would be that this could be a powerful handle on what's, you know, on my performance downstream, this routing function, especially also making this different during inference, you know, any any number of things, doing a Monte Carlo tree search at inference time to be as accurate as possible, kind of like, like AlphaGo or something. Do you have an idea on what the power of the routing function in these sparse models is? And how does it work currently? Like, what's the most, the latest and greatest? And how good is it? Yeah, so this is a really good question, actually, and something we've actually spent a lot of time about. So I would say actually, in this project, probably the thing I maybe spent the most time with is trying out different routing algorithms and routing parameterizations. But we ended up kind of going with the default thing, which I also think says something a little bit about the results of it. Yeah, so I would say my intuition is that the model actually works surprisingly well with a lot of different ways you can route the tokens. So like, you know, we tried a lot of other routing algorithms, we tried making like the routing network larger, we tried like, you know, some fancier ways of actually figuring out where you should send the token to, we tried, you know, using additional information of like, oh, when you're routing this current representation, you have access to whether or not like it was routed, or like where it was routed before in previous layers, using like word embedding information too. But yeah, I think overall, it seemed to be, you know, kind of insensitive, we actually did find like one or two methods that improve things, but they can only be used in certain situations. So it was a bit trickier to just like replace everything. The current routing algorithm we're using is basically what the original one was doing, I think in Shazir et al in 2017, when these kind of things were like really introduced into the LSTM language models. And I think, you know, our newer work, and then also Glam as well, we're using these kind of routing algorithms too. Yeah, and also like one kind of like detail here, it's like, so right now, we're sort of splitting out this little box, and we're like, oh, this is the router. It's not really an accurate characterization. It's like, yes, okay, you're mapping some vector into a vector that has like the same like length as number of experts. But if you just don't update that matrix, it still works fine, right? Because now just the represent like the weight matrices below you are just sort of adapting and just piping whatever activation they need, right? If you freeze the great if you stop a gradient through that, then it's like catastrophically bad. But yeah, I mean, I've also sort of been surprised by the relative insensitivity to the routing algorithm. Like we've seen like, you know, maybe some small boosts here and there, but it hasn't been super significant. I think you probably have a better sort of like a bigger significance by actually just sort of fundamentally changing like the architecture. Like maybe there's like some wildly different approach for sort of sparse models that we're not considering, maybe we're in some sort of like local men. And like these small tweaks on like, oh, okay, precisely, how are we doing this? Maybe doesn't matter as much. And DeepMinds also explored some other kind of interesting routing algorithms, like you sort of alluded to fixed routing algorithms, where it's just like, you're not even learning. They've also tried RL based routing algorithms. And I think it had like actually similar scaling properties. So again, kind of corroborating what Barrett is saying, it's just like, a lot of these things when we're kind of doing this, like per token routing, haven't really moved the needle substantially. That's been our our luck. Yeah, and I think another important trend actually, is that we when we were experimenting with a lot of these different routing algorithms, we actually found that they did help models. And maybe when you had like a 1 billion parameter dense modelish size, but then like, as we scaled up the models, like actually a lot of the time, sometimes the differences would just like wash away, as well. So it's kind of this interesting effect of when more scale is increased, like it maybe becomes a little bit less insensitive to some of these decisions. Yeah, I was I was Yeah, I can totally see that that essentially that the rest of the network adjusts, especially if everything is trainable. What I would be excited about maybe is is to somehow at inference time doing something smarter than because at training time, I can adjust to everything right, but at inference time, maybe there's something that I could do, especially with regards to, you know, domain shift, domain adaptation, anything like this, where I could, I could tweak routing in some way, but I guess that's also, also up for for future work. Okay. So there's a little bit of this not tweaking the routing algorithm, but tweaking the capacity factor hyper parameter I mentioned a while ago. So that's this is basically the parameter that's going to dictate how many tokens are being dropped. And one cool thing you can do is you can have some capacity factor during training. But then at eval time, depending on if you want to use more or less compute, you can be either dropping more or less tokens, and either kind of, you know, increase or decrease the performance, which is pretty cool. And the model is actually pretty robust to having that train from training evaluation time. So that's actually kind of like a good lever for like, you know, depending on if you want to use more or less compute during evaluation. I think we have we have a pretty good overview. Now I want to get a little bit into just the future, the future prospects, maybe also of this we already talked about, and with pathways, we could have heterogeneous things, could this be pushed to some sort of limit? Whenever I see a distributed system, you know, I immediately think distributed maybe not even in a data center, but across users across networks is their applications to maybe, what was it called federated, some kind of some kind of federated computing, some kind of federated learning where I could somehow contribute with my maybe confidential data, but I could still contribute to a whole compute process is there, I'm gonna say the the B word, is there an application for blockchain distribution, something like this? Like, have you do you think about sort of the higher degrees of distribution here? Do you want me to go for it? Yeah, go for it. I mean, yeah, so I mean, yes, me personally, I haven't spent a ton of time thinking about this. But I do think it's like very interesting. And yeah, there definitely seems to be a lot of really, you know, open problems around this, especially given the growing amount of like fragmented compute, fragmented devices, like there's so much computer on here, like, you know, how can you effectively utilize all of this, utilize different, you know, data and stuff, I think it's like a super cool and I think it was going to require a lot of really interesting research, because right now the way we're currently training these models is it's all like synchronized lockstep typically, right, you're doing like, oh, like after each batch, you do these gradients, you send the gradients around and everything. But like, I think actually, maybe the future of these models, when you're really, you know, allowing them to be distributed across very different types of computing, everything might actually now introduce like asynchronous training as kind of like the new paradigm. So I think that's like a really exciting space. But yeah, I haven't spent too much time thinking about it personally. Yeah, and I think like, as it pertains to say like blockchain or something, like, I think one problem with these expert models as designed in this way, are these all to all communications. So over this sort of like, you know, decentralized, like peer to peer network, where it's like, you know, nodes are like, you know, really far apart, inconsistent sort of bandwidth and stuff. That could be really tough if sort of your experts were sort of distributed among like many different nodes in this sort of like unreliable network where nodes are kind of coming and going. Like right now, all our systems are in this sort of like very constrained fault intolerant area where it's like, oh, all highly internet work chips that are highly reliable. And then so like blockchain would just have like a whole different set of like kind of problems that you'd have to sort of address like unreliability and you know, some of these other areas, not to say I think you just like require some like additional kind of research, like just sort of adopting the model as is, I think would pretty poorly map on that kind of computing infrastructure. But I think there's something there that could be done. Is there work on because I see these works mostly here in NLP yet transformers kind of taking over the rest of the world. Is there work on how these experts, sparse expert transformers behave in vision in reinforcement learning, speech, whatever? Yeah, yeah, great question. So absolutely, actually, there's been some really good work applying these models to like VIP based, like image classification and stuff. And there, it's actually really nice, because then you can leverage all of the, you know, niceties around like people figuring out how to get these working really well and transformers and kind of, you know, nicely map it over as well. I've, yeah, there's also been some good work using these in speech as well. Liam, any other things to add on top of that? Some, I used to do reinforcement learning more full time, and some colleagues kind of reached out about doing like sparse expert models for RL. I haven't seen, I'm not familiar with some work. But, you know, that might be sort of like another interesting avenue, but like for sure. So language, vision, speech. I don't know if there's been any videos, any video work yet. Yeah, but like high data, a lot of throughput, those would be like, you know, really good areas. So I think video would be also really promising. Yeah, I really like also the, I feel like it feels very natural in these high dimensionality spaces that you really might want different parameters to be applied, like when you have a video, like one, I think you don't want to be applying the same amount of compute to every frame. But then on top of that, I could see like, actually, you really want to have different parameters applying to different, you know, things going on in the video, because it's just gonna be like wildly different stuff happening. So yeah, I think I'm very excited about these models for video as well. Do you imagine that these models will just, essentially right now they're competition to dense models. They are competing, you're tracking Pareto frontiers, how much compute, how well are they doing, tackling very much the same tasks. Do you think this will go on? Like, do you think these models might overtake dense models if we figure out how to handle them correctly? Or is it more like there's a killer app for each one of them? Yeah, I think in, oh, do you want to go ahead, then? Yeah, I mean, I honestly think that the future is going to be adaptive. Like, I don't think there's any way that like in 10 years, our models are treating all examples coming in with like the same parameters over and over again, and the same amount of compute. It may not be this precise sort of like sparsity regime, or may not be the precise sort of adaptive computation, kind of like paradigms that have been put forth. But I view this sort of kind of work of like sparsity adaptive computation, as kind of like inevitable, like, I don't think it's going to be considered like competition, it's just going to be sort of like integrated into a lot of like leading models. That's, that's my expectation. I'd be really shocked in like 10 years, we're training like a 100 trillion parameter dense model. And it's just kind of doing the same thing, like, over and over again, for no matter what comes in, just seems really strange to me. What's the future for your particular research? Like, where do you see, where do you see yourself going in the next, maybe not the next paper that you haven't published yet, but maybe a bit broader time scale? Like, what excites you? And what are your next plans here? Yeah, great question. I mean, I think the thing that really excites me is like what we were kind of talking about earlier of each input getting a different amount of compute applied. Like, I think right now, the models are working well for each input getting different parameters. And I think, you know, coupling this with like adaptive amounts of computation is like, I think, really where I want to be spending time thinking about in the next, you know, upcoming years. Is there? Yeah, I don't know, is you have something like Ponder, there's PonderNet, and so on, there's these recursive architectures, or recurrent architectures that, that sort of decide themselves when to when to exit. Would that be one thing? Or do you simply imagine that each expert is kind of one is the buff expert, and one is the lean expert, and then the routing function essentially takes care of the different amount of compute? Yeah, I don't know. This is a great question. I think, I don't know, I can see either approach potentially working, or maybe you actually want combinations or potentially something completely new. Yeah, it feels like the space is still, you know, very exciting. And there's like a lot of really interesting different verticals being pushed. So the space still feels like, you know, pretty young to me. Okay, last question from my side, what's the connection of this to something like Capsules? I don't know if you've ever thought about the the connection there. But with Capsules, I always think this is these abstract, very abstract things, very high level ideas flying around. And you here have something like very practical, you know, very on the metal thing. Yeah, there seems to be quite some commonalities. Is there is that something that ever came up to you? Or, or is that something that ever came up to you or? In the two years of doing sparsity research, this is literally the first time. I actually should be going back to that work. I feel like capsules like yeah, had a lot of like really interesting conceptions. But maybe like you're kind of alluding to it didn't like map super well to the metal. So maybe that sort of like hindered it's like it's use, whereas this is just like highly motivated from like an engineering perspective. We've had like some questions like, oh, what is like the neuroscientific kind of motivation of our work? And it's like, it's really engineering kind of driven. So it's like, okay, what what will be fast on our existing hardware? But yeah, I will revisit capsules and kind of see like, oh, okay, how could we actually map this a little bit better to the hardware? And like, you know, I think that could be like, you know, an interesting source of ideas. Is there any any last thing you want to get out to viewers that they should take away from this work? Any any way that a regular person can get into this type of research? Anything like this? Yes, a great question. So actually, one thing we tried to show in our switch transformer work is that these models work pretty well, even if you only have two experts. So I definitely I don't want people to think that, you know, you're really a supercomputer to run the models or to, you know, get benefits from having experts, even having I think, as little as two experts and running models could lead to developing really interesting research ideas, improving the performance and everything like that. So yeah, I definitely hope that, you know, more people can continue to experiment and push forward these models. Yeah, and then I would say, like, another interesting trend that I've been following is sort of in parallel to sparsity in these like, you know, really large models is the idea of like, well, what if we just sort of like, have the model sort of offload and like, sort of do lookups or, you know, look at documents and retrieval type methods. I think this is sort of like a very interesting area. And I'd love to see like, kind of head to head comparisons of like, okay, do we want to try to encapsulate the knowledge into parameters? Or do we want to just like, keep it sort of like, you know, parametric, non-parametric type thing? And we keep the information kind of written in docs or like, what does the interplay look like? I think that's sort of like another really interesting avenue, like, kind of comparing these things. Awesome. Yeah, it sounds really cool. I'm excited to to see what the future of these models bring. Yeah, Barrett and William, thank you so much for being here. This was a lot of fun. I hope to see you again soon. Yeah, cool. Thanks for having us. Yeah, thanks for having us.
 Hello, today I'm having an interview about the topic of sparse experts. Now, ironically,
 the people are absolute experts in this type of models. These models, they are huge, they're
 usually language models, but they don't have to be they're usually transformers, but they don't
 have to be what they do have in common is this notion of sparse experts, these models go up to
 the trillions of parameters, and they achieve this via sparsity. Now I want to do a very,
 very brief introduction of what sparse expert models are. And then we'll dive into the interview
 right away because I don't want to keep it from you. So let's look at a transformer model. Usually,
 I have some sort of an input that is tokens, a sequence of tokens, which are represented here
 by circles. And what I'm going to do with these tokens is I'm going to alternatingly push them
 through different layers. Now one big layer type that is common in transformers is the attention
 layer, we're not going to talk about the attention layer today, all you have to know is that it takes
 in a sequence of tokens, and it outputs a sequence of tokens again, ideally the same amount as went
 in, which I failed to draw here, the other very common big type of layer in these transformers
 is what's called the feed forward layer. Now the feed forward layer is just a linear layer,
 and every token goes through this linear layer by itself. So every token individually goes through
 the same transformation. And thus, as we do this with all tokens, again, we end up with a sequence
 of as many tokens as we input. Now a sparse expert model isn't very different than this,
 the attention layers commonly aren't really touched. So that works just the same. However,
 in the feed forward layer, we see a big difference. Notably, we don't only have one feed forward layer,
 we have many. So here is feed forward one, here is feed forward two, here is feed forward three,
 and here is feed forward four, each one representing a different individual linear
 transformation of a token. Now when we talk about sparse experts, these things here are called the
 experts, they're called the experts because they're thought to specialize in very specific tasks. And
 the goal in sparse expert models is to route the tokens to the corresponding correct experts. So
 every token goes through what's known as a routing function. We're going to talk about this routing
 function in the interview. But in essence, it is a very simple, usually something like a linear
 function or a simple transformation that decides to which of the experts any given token is routed.
 So sometimes even in sparse expert models, a token is routed to multiple experts. But in the newest
 iterations, the tokens are simply routed to one single experts and none of the other. Usually this
 is done, as I said, by some sort of a linear transformation, followed by a softmax to decide
 where the token goes. So every token would be assigned to one expert. And that gives the
 possibility of scaling these models up dramatically. Not only do you save a lot of compute because the
 tokens only go to one place ergo, you only need to compute that one thing for that particular
 token. But also there's the opportunity to massively shard and parallelize these different experts
 across different machines, as you only need to route the token to one place. That means you
 dramatically reduce these big all to all reductions, they still happen, but not as much. So as I
 already said, the biggest models have trillions of parameters, you need to take a little bit of care
 of how you then aggregate the tokens once they come out of the experts. So essentially what you
 want to do is you want to carry over the likelihood from the routing function up here. But this is a
 minor detail, a minor details are important, but you know, so I know it doesn't look like much,
 but these sparse expert models really have the potential to massively scale up our current
 efforts in AI. And I have no doubt that they're going to play a role in the near future, when
 we're looking at bigger and bigger models, because at some point, the purely dense models will reach
 sort of the limit of what's physically doable. And then it's a good opportunity that we have models
 that can go even larger. Alright, so without further ado, let's jump into the interview. I hope you're
 enjoying yourself. If you do have any sort of comments, please leave a comment, share the
 video around if you like it, and I'll see you around. Bye bye. Hello, everyone, my guests today
 are William Fedes and Barrett Zoff, who are engineers and researchers at Google, Google Brain,
 and have been diving into large models, specifically sparse expert models, which are models that,
 well, feature this notion of experts, and also have a notion of sparsity. And hopefully today,
 we'll discover what this is all about. Specifically, we'll talk broadly about three papers in a long
 line of work. One is the switch transformers paper, which was really, I believe, one of the first
 papers that just had like massive amounts of parameter was that like trillion, probably
 trillion parameters. It was big. 1.6 trillion parameters. That's right. Yeah, yeah, it's insane.
 And then there's there's glam, which demonstrated really nice scaling laws with these sparse experts.
 And more recently, there is designing effective sparse expert models, which as far as I can see,
 is also a bit of a of a maybe a summary recommendations, more of a what we learned
 type of thing. So William and Barrett, welcome to the channel. Thanks so much for being here.
 Yeah, thanks for having me. So can you give us just a little bit of context what you mean when
 you say sparse expert models? Yeah, sure. So this is a great question, especially since the word
 sparsity crops up in like many different aspects of deep learning, whether it's, you know, like
 sparse attention or, you know, various other sparse paradigms. So yes, sparsity in our case
 means that each input can get different subsets of parameters. So that's kind of like the main
 sparsity that we're talking about here. And it's like, you know, it's a very natural concept,
 right? Like normally, in like a dense transformer, for example, you have, you know, a word embedding,
 and, you know, any word will have the same parameters and compute applied to it. And in
 sparse models, typically what happens is you have the same amount of compute, but you can have
 different subsets of the model parameters be like, you know, acting on the model inputs.
 And what does that mean in in practice? So we're talking mainly about, let's say transformer
 models here. No, is that a good characterization of things? Or do you do you see sparse expert
 models in a more general sense? Yeah, I mean, these things actually almost sort of like cropped
 up originally as almost like in the context of like ensemble type methods, where you have
 a bunch of like almost like fully independent models. And then you're sort of using these as
 like, you know, each model as an expert. But the common paradigm as of like 2022, is sort of
 experts as a layer. So this is like really popularized by Noam Shazir's work in 2017,
 outrageously large models. And in that context, they were actually inserting it in between LSTM
 layers, which is like the prevailing like recurrent architecture at the time. Most of the things just
 because like the world has sort of shifted towards transformers in it seems like almost all modalities
 now, we're often thinking about experts as a layer inside transformers. Typically, we're sort of
 doing this at the feed forward. So these blocks that just sort of independently apply on the
 different like tokens. But we've also kind of considered it in self attention layers, it's
 just sort of like a very general concept. But yeah, typically in transformers. So you have this
 notion of an expert, which you say is is sort of a specialized function or something like this. And
 then there's often this thing called a router. How how does information find its way through these
 experts? What are the general principles in that? And why would I even consider doing something like
 this? Yeah, so great question. So yeah, so you have this figure up here. And so one thing to
 notice that basically, if you only have a single expert, it essentially reduces to just a normal
 dense transformer. So the interpretation is pretty natural. And in almost all of the ways people are
 doing sparse expert model nowadays, there's some notion of a learned mechanism that for, you know,
 embedding at the current layer, you figure out what expert you should send this representation to.
 And this can be ranging from very simple to just like a simple softmax function over the
 total number of experts to very complicated linear programming type solutions that have a more
 like globally optimal solution. So yeah, so this is this is kind of like the paradigm. And I think
 it's a pretty natural one. So even if you want to only, you know, yeah, apply one set of weights per
 representation, now you have the option of just instead of always applying the same weight matrix.
 Now you can, you know, maybe have a selection of in this figure for different weight matrices. And
 the way that, you know, we've done this in our work, and I think is the most common is just as a
 single feed forward network. So you take your input representation, and then you just, you know,
 apply it with something that's going to be like, you know, the model dimension by the number of
 experts, and then you apply like a softmax function to get like a probability over all of the different
 experts. And our switch transformer work, the routing was extremely simple, where it's just like
 you just send it to the highest, like the highest expert with the highest probability. And then, you
 know, you just simply route it to that expert, then the output of that computation gets scaled
 by the router probability. So if it was like, oh, with 0.9, send it to expert two, then when you
 have the output of that computation, you scale it all by 0.9. Do I remember correctly that there was
 some paper that it was this an older paper, and this might be getting very technical for a second,
 but was there an older paper that said something like you always needed to send it to at least
 two of these experts, otherwise, it's kind of unstable. Is that an older paper, or a newer
 than yours? It actually wasn't instability that they're clashing against. It was more this idea
 that we're doing this like weird discretize operation. So instead of using like reinforcement
 learning to sort of like update on the experts, we're kind of doing this like kind of hacky back
 propagation through these like softmax operations, which have been masked. And the idea that top two
 or greater was necessary because they were thinking, well, I'm creating a probability
 distribution for this token for this word over the available experts. If I don't have at least two,
 I can't tell whether expert i or j was sort of better for this one. So it's like, in order to
 have the hypothesis was sort of like a useful gradient signal for the router, it has to know,
 well, should I have sent it to i or j? And then we just sort of didn't follow convention and did
 one. And it also seems to work just fine. I think in part because you're sort of doing this sort of
 normalization. So you can still get an up waiting or down waiting if you select an expert. So it's
 like, oh, if that expert selection worked out well for you, or worked out poorly for you, you can then
 sort of adjust the embedding for that expert. And then you at the next pass, if you saw that same
 token, you're still doing this like softmax distribution. So you're kind of like up waiting
 or down waiting it. So I think that's sort of like the gist of the mechanism. And this, this, I think
 this idea was at least from 2017, it may have predated it. Could you maybe now that we're talking
 about history, trace the evolution of this line of research a little bit. You already mentioned this
 existed as sort of ensemble methods inside of it. I'm talking now specifically about sparse experts
 within transformers, which are the things that allow us to really scale up to these giant models.
 What's the what's sort of the line of research? What are the original things? I'm going to guess
 this this work is among them. And what were the improvements that happened since then in this field?
 Bear, do you want me to go or you go for it? Yeah, so I mean, like, going back 30 years, like you have
 like Jordans and Jacob, this obviously predates transformer because transformer was a 2017
 development. So I mean, the concept is very, very old. I think it just kind of like resurged in
 popularity. I'd say the first, yeah, the very first sort of use of mixture of experts in
 transformer was left in it all in 2020. So this is G shard. And it just showed really remarkable
 improvements in translation. What they were doing was, you know, analogous to switch transforming
 these other works is they just sort of substitute these feed forward blocks with experts. And in
 that case, sort of also similar with switch transformer, they had many, many experts, I think
 in that case, it was thousands. And they were showing really significant improvements over
 state of the art translation models. I think as the field has sort of evolved, as we've sort of
 like learned a bit more about it, there seemed to be this like kind of general trend of like,
 okay, cool, we can pre train these models or like in the case of translation, there's no big
 distribution shift. When you're training to translate, you're also doing inference to translate.
 But in switch transformer, we found, okay, we'll pre train to, you know, improve the perplexity,
 improve the prediction of next token. And we were getting significant improvements. But then when we
 took it under a data distribution shift to fine tuning, it was performing quite badly with many
 experts. So I think there's been this trend to try to balance the computation and the parameters a
 bit more. So I think some of the prevailing models have actually in transformers have actually gone
 have actually gone towards fewer experts. So 1632, 64 experts, not 1000s of experts. So that's kind
 of like the lineage of mixture of experts and then like mixture of experts in the context of transformers.
 And what is so in that context, if one expert is the classic transformer model, and that seems to
 not work as well as many experts, but too many don't work, what is the abstraction that I can
 think of for an expert? Like, what does an expert learn? What is an expert responsible for?
 Approximately? Do you have any idea what happens? Like what, what, how does it make sense that the
 optimal number is, let's say, a few dozen and not super many, but also not one?
 Yeah, so great question. So yeah, there's like a few parts to this. So one, like, I think it's
 really just like an empirical observation right now that, you know, 16 versus 64 versus, you know,
 2048 versus 10,000. You know, like, it seems like the expert numbers in the middle, like,
 it's not from the standpoint of like on a per step basis, more experts typically don't make things
 worse. Usually it's like better or about the same, but things start to level off. But it's
 very inconvenient to have a lot of experts because it's just this like a huge memory footprint,
 the way that the models are distributed, it's not really amenable towards typically, unless you have
 like tons of, you know, parallel cores going. So like actually the observation where you kind of
 want to actually have like a middle amount of experts is a lot of the times actually driven by
 just the like practicality of then like training, serving these models. Yeah, in terms of like,
 what these models are actually learning, like intuitively. So we actually studied this in our
 most recent work, kind of looking at, you know, each expert, what are they specializing in, what
 are they learning? And interestingly, they kind of specialize in some shallow concepts, which you
 would think maybe there would be like only really deep things going on. And it would be kind of hard
 to inspect them. But you know, we noticed like, oh, there's like a punctuation expert, or an expert
 that will, you know, talk about, you know, like proper nouns, which we thought was pretty funny,
 and maybe not super intuitive for, you know, how. Yeah, actually, if you want, you can switch over to
 the recent paper, and we actually have a figure which sort of shows some of these things. So you
 can kind of like follow along and see how shallow these things actually are. Yeah. Yeah. So this
 this would be this would be different. So you you found an expert or in this case, multiple experts
 that that focused on the these sort of things. So there's conjunctions, punctuation, verb,
 visual description, which is which is interesting, because that's kind of I want to say like a higher
 level thing than just the punctuation, right? Counting numbers. Yeah, how do you make sense of
 this stuff? Like, what's going on?
 I Yeah, I mean, I think we were sort of expecting maybe like a higher level of description, but like,
 or like, sort of like representation. Um, it's, I think we've just started started to sort of like
 crack and like, look into these models to actually see what's going on. That obviously, like one big
 specialization that you're seeing here are these Sentinel tokens. To make sense of that, we were
 sort of doing pre training where it's sort of fill in the blank test. And a blank is sort of represented
 by these like little Sentinels. So like extra ID 10 represents like, you know, the blank 10. And we
 often really frequently see experts are specializing on these blanks. So, like, we're
 doing pre training. So that's sort of an interesting thing. And then I think that also might segue into
 maybe you want to actually, given this sort of like, you know, observed specialization, maybe you
 actually want to make some experts higher capacity or give them more compute to sort of do things
 that might be harder. But honestly, I mean, this is still very early. It'd be interesting for sort
 of like, you know, some of the interpretability lens that like entropic has on some of the recent
 sparse expert models. Some questions we've kind of received are, what is the interplay of expert
 specialization with sort of like self attention specialization? And that's honestly completely
 open. I think we were just sort of putting this table forth to the community to be like, well,
 we, we started, it's not exactly what we would have expected. But definitely kind of like a call to
 dig further and hopefully like, you know, further improve things with the also I believe that this
 was Oh, yeah, here already in switch transformers, this ability to distribute these things across
 devices that comes naturally with, with having sparse experts. So sparsity meaning in this case,
 I only send stuff to one or a few experts. And there there came the ability to shard this across
 devices, how, like, how practical is this really to like, what, when would I do something like this?
 At what point would it become practical and useful and the best thing to do to communicate
 across devices for my experts? Yeah, so really great question. And I actually think this is
 the reason why the method works so well, actually. So the standard way I would say people are doing
 distributed training of these models is they have, you know, either fully data parallelism, which
 means like, you know, each machine has the same set of weights, but different slices of data, or a
 blend of data and model parallelism, where it's like, you know, kind of a mix where certain like,
 you know, cores have sometimes different weights or sometimes different data, and then you communicate
 stuff to make it, you know, emulate like a full model. But I think experts, one really easy
 interpretation of this is like, let's say you have a model, and, you know, you're using data parallelism,
 and you have four different machines, a really natural way to overlay experts on this would be
 you just have one expert per machine. And then, yeah, so this is like a really nice interpretation,
 because then when you have all of your, you know, local data per core, you'd have the router weights
 replicated, but then you just figure out what expert they need to go to. And then that's when
 you kind of, you know, shuffle all the tokens around to the machines, do all the computation,
 and then shuffle them back. And this makes it really nice, because then per machine, you actually
 never have any more parameters than you would have had just with the Dense Transformer. But now you
 have experts. So it's actually like a really nice way of kind of, you know, thinking about how to
 design the models would be like, oh, you know, you have this many cores for data parallelism,
 just have that many experts. And that's actually a paradigm that Bim and I use a lot when designing
 these models as well. And yeah, I mean, I think as soon as you have this sort of like, distributed
 model, where you're already going across accelerators and devices, you do already have
 these communication patterns, right? Like you need to get activations to a certain place, you need to
 like get gradients to a certain place. So you already have these sort of like all reduced
 communication collectives. Expert model is going to introduce all to all communication patterns. So
 that can be like a more expensive thing, especially based on like your topology and
 the bandwidth between all of your networks, or between all of your devices. But yeah, so I mean,
 this is something you sort of have to like, kind of empirically test like, okay, how much does this
 architecture kind of buy you in terms of performance on your task, versus the additional
 costs of all to all communication. But you will be communicating across devices for these big models,
 regardless to train them. Yeah. So this is a good, I guess, a good segue, because you can achieve
 these giant models, like trillions of parameters using these is the sparse expert models, because
 naturally, I can parallelize these experts, it doesn't cost me really much more compute,
 because any data point, or any token only goes to one single expert. There is always a bit of the,
 let's say, the question of how comparable this is to the dense models. It was it was often I don't
 know if this is a latent feeling that I get from the community, but people would rather have the
 175 billion GPT three model compared to the switch transformer, even if it is trillions of parameters.
 Is there some sort of division factor where I could compare to a dense model? Or do you think
 that it's an entirely different nature of function that's computed here?
 Yeah, so this is a really great question. And I think there's a lot of different ways you
 have to kind of look at this to figure out if a sparse model is right for you.
 So I think actually, in a lot of applications, if it's like, hey, I want to train the model
 with the smallest memory footprint, so I can just be using it on the smallest amount of
 devices as possible, a dense model will always be better. Like I think on a per parameter basis,
 dense models are going to be performing better. So for those types of applications, I'm like,
 yeah, I don't think it makes sense to be using sparse models. Maybe you want to just train the
 best thing that you can fit onto your local 2 GPU machine or like a 10 GPU machine, and do really
 kind of low throughput, feeding in data to this, like not high or anything like that.
 I think sparse models are good, where you're going to be training a model and you're going
 to be hosting it on a lot of machines and you're going to be having a lot of high throughput going
 through it. So a lot of queries, a lot of stuff going through it, because then things can be
 batched together and then the models actually become pretty efficient. So I think that's kind
 of one lens to look at when you would want to use a sparse versus dense model. And I think the kind
 of second lens is that, for a given amount of GPU or TPU hours on a compute cluster, what model
 will get you the best performance? And I think that's the lens that we actually would spend a
 lot of time looking at for pre-training models in this paper, like, oh, you have 512 TPU chips,
 and I give you X budget training hours, is a dense model or sparse model going to give you
 the best pre-training performance? And I think our assessment was that, yeah, I think actually
 the Pareto optimal model typically is a sparse model in that setup.
 Yeah, and comparing parameters, especially between a dense and a sparse model, is just
 totally incomparable. So using GPT-3 and then our largest switch transformer model,
 it's just wildly different amount of computes in our case. You can't infer that from the parameter
 budget. So I don't know what the compute ratio was between the two, but far different. Our 1.6
 trillion parameter model was actually only doing about as much compute as a billion parameter
 model. So for each token, it was doing roughly a billion parameters worth of flops. And whereas
 GPT-3 is doing 175 billion parameters worth of flops. So you can sort of tune this, and DeepMind
 has sort of also tried to come up with a characterization of scaling properties, far more
 robust than we've been able to do, of sparse expert models, and try to come up with a dense
 model equivalent. So that might be an interesting work to refer to in the future. But really,
 it's just like, practically speaking, it's like, OK, I give you these accelerators for this amount
 of time. What's the best model? So that's probably the fairest comparison.
 Have you seen this Pathways paper?
 Yes, definitely.
 They came out. How does it play into something like this? Is it going to make this easier? Is
 it going to make it superfluous? How does the ability to schedule things heterogeneously across
 devices, or does it enable new possibilities in the sparse expert world?
 Yeah, so great question. So one thing to note is, OK, so typically you have dense models. And a
 dense model, like every input, will have the same amount of compute and parameters applied to it.
 And sparse models, now you have the same amount of compute, but different parameters.
 And I think the kind of natural next step that I think makes a lot of sense to both Liam and I is
 that now for each input, you have a different amount of compute applied as well. And I think
 Pathways is really exciting, again, like you kind of mentioned for the heterogeneous compute,
 where we want to have inputs that might require different parameters and also different amounts
 of compute. Yeah, and I think a framework like this is going to really open up a lot of really
 exciting research avenues along that direction. And I think it feels like a very natural
 interpretation for kind of where our models are headed for in the future.
 Yeah, like right now, it's like our experts are all sort of completely homogenous. They're all
 the same size. They do the same operations. Pathways, you could be like, oh, this is like
 a recurrent expert. This is a huge expert. There's a group of small experts. You could just be
 a lot more flexible in design. And sort of like alluding to that a little bit with when we were
 sort of looking at the visualization, it's like, oh, wow, a really consistent thing. Our experts
 that want to specialize in these like fill in the blank tokens, these Sentinel tokens, perhaps that
 might be an avenue or an area where it's like, oh, let's dramatically increase the compute here.
 This is, oh, hi, Kat. This is like an area where we like a lot of extra compute could really be
 helpful. And there wasn't really an effective way to do this with the existing infrastructures
 before pathways. Is there a... Yeah, sorry, that's lost the train of thought. Explain to me a little
 bit how GLAM improved upon switch transformers. Like what's new? What's exciting there?
 Yeah, so I think GLAM... So one also thing to note is like there's kind of a right now division of
 two different types of model classes in language modeling space, I would say. So one is like these
 decoder only models where it's just a single set of parameters and it's like you're just predicting
 the next token like autoregressively. And this is like what GPT-3 is. And this is also the kind
 of architecture that GLAM studies these models in. So the other classes, these encoder decoder
 models like T5, this was also G-shard. This is kind of what also we studied in switch transformer
 in our most recent work as well. So I think GLAM did a few things. So one, they really, I think,
 pushed the scale of these models. So like while our original model of switch transformer had more
 parameters, like GLAM had like much more compute applied per token. And they studied these very
 extensively with decoder only language models. And yeah, I think their main comparison point was
 to GPT-3 as well. So they were studying a lot in the context of few-shot and like one-shot evaluations,
 whereas I think a lot of our work actually centered around like fine tuning the models.
 But yeah, I think GLAM really like pushed the scale of these, especially in these decoder only
 language models and showed that like, yeah, you know, you can get as good of quality as GPT-3 with
 like, you know, huge computational training savings as well. And they did a really, a lot of really
 good work in that space. Is there a functional difference between the sparse expert routing or
 anything around this in GLAM? Or is it mainly what you said with decoder only and applying
 more compute scaling it up? So actually, there is a few differences that are more nuanced and
 technical. But yeah, at a high level, you know, there's a routing function, and they actually
 route each token to two experts. And actually, there's like some of the differences in these
 models comes from like how much buffer you give each token, each expert, because, you know, you
 need to have like fixed batch sizes for all the experts ahead of time. And so what can happen is
 like, you can't guarantee that like, there's going to be perfect balancing among all of the tokens
 getting sent to experts. So like experts can overflow. And there's this key parameter that
 we call the capacity factor. That's probably the single-handedly most important parameter when
 designing a mixture of expert models, because it just has such a huge impact on the communication
 costs, compute and everything like that for how much buffer you should have. And yeah, I think
 a big difference from GLAM versus our models is they actually use like a much larger capacity factor
 than we've used in our other works. But yeah, the routing algorithm is essentially the same.
 That is, yeah, I want to get a bit more into the into the routing algorithm in just a bit,
 but just to end this with the with the last paper that we've previously looked at, was I right in
 saying that this is much more often, let's say a general, almost like a review paper? Or how would
 you describe it? Yeah, I mean, I think we we tried to make sure like we're contextualizing a lot of
 the work. So we tried to make sure the related work was like, pretty inclusive, because I mean,
 I think the field's really adjusted and improved a lot in the last two years. But I would sort of
 characterize this paper as fixing the two big flaws from our first one from switch transformers.
 The first was these models are unstable to train. So we'd be training and then all of a sudden the
 loss or just diverge, which thwarted a lot of our issues. Interestingly, it doesn't seem like the
 instability arises from a lot of experts. We were consistently able to train models like our
 trillion parameter model, for instance, with thousands of experts, never really hitting any
 unstable sections, really kind of came from like high clops or high computation expert models,
 even with like few experts, those were highly unstable. And then the second thing that this
 paper sort of fixed was the sort of like poor fine tuning quality. So we would sort of pre train a
 model, it would show like really significant speed ups over a dense counterpart. But then when
 it came time to fine tuning, say I'm like super glue or some like other task of interest, it would
 just be considerably worse. So I think this paper was just really trying to sort of like kind of
 patch up a couple of those issues, we identified them in our first work. Yeah, I'm always a bit
 intimidated when a paper has a table of index by itself. Can you can you go to something that
 Barry and I discussed, it's like, okay, should we break this up into multiple papers? Or should this
 be one because, you know, this is like, you know, a lot of work. And, you know, this is like something
 that we discussed, like maybe in the future, we should probably be producing like more bite size
 pieces of work. When you when you talk about fine tuning, can you go a bit into more detail? Like,
 what was exactly the problem? How did you how did you also go about fixing it? So I'm not only
 interested in, you know, how did how what's the final model like, but what does the process of
 debugging something like this and then getting to an architecture or a solution that actually works
 look like? Yeah, I mean, it's sort of this like very interesting problem of like, you want to,
 there's really just like fundamental trade off. And whenever you're sort of doing a sort of like
 large scale work, where you want to try to understand and characterize things at a smaller
 scale, understand scaling properties, understand, understand like hyper parameter dependencies.
 But then you also want to be consistently checking yourself at the largest scales. And this sort of
 balance of like, okay, you have this much compute, you have this much time, where do you allocate it?
 Do you do a lot of small experiments? Or do you do a few big experiments? It's kind of tricky.
 But I'd say part of our like findings were the first one was like, okay, well, characterization
 is we're not doing better on fine tuning. What's the cause? And it seemed like perhaps our cause is
 not that of optimization, it's that of generalization. So if you scroll down into section four,
 you can just click on the link. We might be Yeah, exactly. Yeah, so this is an example that, you
 know, kind of supports a lot of the trends we're seeing. On the left is a small superglue task. So
 this task has only 250 training sequences, so very small. And on the right is record. So this has
 over 100,000 training examples. We're showing sparse models versus dense models in the two things,
 in the two plots. Blue represents the sparse training though, and you can see it just very
 quickly gets to 100%. And it outpaces in both cases, the small task and the large task outpaces
 the dense model getting to 100% train evaluation accuracy. But when in the small task, we'll see
 the dense model in red actually outperforming the ultimate performance for the sparse model in orange,
 whereas for the bigger tasks, the sparse model does well. And so we kind of kept seeing this like,
 you know, overfitting issues. And a lot of this was then led us to sort of like investigate
 hyperparameters. And, you know, some of the hyperparameters can sort of be adjusted in a way
 to make the model like less susceptible to overfitting. So you can use like different
 dropout parameterizations, but also things like batch size and learning rate can inject more noise,
 which can also be sort of like a counter to some like overfitting properties. So we tried and then
 sort of consistent with this, like a lot of these things were sort of like, you know, more exhaustive
 studies at say, a billion parameter scale, we then tried to continue to sort of like fact check this
 against our larger model, and make sure that these conclusions were holding. So I think it was just
 sort of like, you know, the debugging process was, okay, what more precisely is going wrong? And then
 like, what are our levers that we can sort of like pull in order to try to like improve it? But you
 know, a bit of art and science really. You so you is it you observed, okay, we are probably overfitting,
 because you saw the smaller the tasks got sort of the worst the sparse models would ultimately
 perform on the validation set of those tasks. Did you? And you have it's not like quite like,
 yeah, it's not always like quite so easy as that. But it's sort of like, you know,
 directionally, like, I think we have support of the hypothesis. But it's not like every single
 small task does poorly. And every large task is great. Yeah, but I'd say directionally,
 it seems to be a phenomenon we've observed. You have also a bunch of experiments down here where
 you where you investigate some of these, for example, dropout probabilities, you also have
 expert dropout probability, which is one of the questions I had in that you have particular
 architecture, right with these with these experts. And when I think about overfitting,
 what in regular transformers, I have kind of handles, I can use adapter layers, I can only
 fine tune the head and so on. Did you ever investigate maybe only fine tuning some of the
 experts? Like is that the keeping others constant? Is that ever a thing? Like, would that work? Or,
 or, or, you know, can we can we make use somehow of the fact that we have these different experts,
 and they're actually different functions? Yeah, great question. And I think actually,
 if you scroll down, we did a very naive kind of version of this, not where we freeze different
 experts, but we, you know, freeze all of the experts, or maybe only train all the experts
 and freeze all of the other parameters. I would say our findings were this were surprising in
 a bad way. So nothing, nothing really worked super well. So here you can see that and this is also,
 we only studied this on super glue, right? So it's far from exhaustive. But yeah, so one thing we
 tried was updating first all of the non mixture of expert parameters only. And that actually
 performed about the same, which was kind of interesting. It's like, hey, like actually
 freezing the mixture of expert weights like seem to perform about as well as just like updating the
 whole model. Then when we started to, you know, update only the mixture of expert weights and
 freeze all the other model parameters, like the performance was actually really bad. And there
 was some we still fully don't understand what's going on here. We have like a few kind of like
 half baked hypotheses. But yeah, then when we update only the attention parameters, things are
 worse. And we found a slight boost updating only the feed forward network parameters that weren't
 the mixture of expert layers. But yeah, overall, nothing worked that well. But yeah, I think there
 might be some potential really interesting things of like, hey, maybe allowing only, you know,
 a certain subset of experts to be fine tuned. We did spend a little bit of time actually studying
 like pruning off experts during fine tuning. So like for a specific fine tuning task, if your
 pre trained model has like 64 experts, can you just take like a subset of like two, four, eight or 16
 of them? Yeah, and we also didn't really get that good of signal with this as well.
 Also to some of your recommendations, they actually would be compatible with expert models too. So
 you're free to just like fine tune like the top, like top logit layer, or you could add in adapter
 layers. Yeah, we didn't do anything like really funky, like you were suggesting like, oh, we're
 only going to expert like update experts like three, eight and 14 or something. Yeah, my intuition
 is that probably wouldn't work well. But I mean, I've been proven wrong many times. Yeah,
 we tried some like other things that didn't make it to this table or these plots. And yeah, again,
 we didn't really see like a significant boost. That said, if you are only updating like a fraction
 of the parameters, you get some memory savings. So you know, some nice things.
 Cool. I guess one, you know, there's, there's almost an infinite number of things one could
 try with these things like distilling experts like distilling multiple experts into a single expert.
 So you have another expert that's again free to do some some new tasks. Once you know that
 two experts are converging something like, I think there's, it's really interesting, right? A lot of
 we're adding a new experts on the fly. Yeah, a lot of possibilities. And that brings me a bit to
 this this routing function that we talked about before and at the beginning, which seems to me
 is a really crucial part of the system. Yet, as you said before, very often, I've just seen this
 being implemented quite simplistically, maybe there's a linear transform and then a softmax
 or something like this, maybe not even maybe there is some some sort of a, you know, a
 some fixed keys for all of the experts and then you route according to that. Do you like my intuition
 would be that this could be a powerful handle on what's, you know, on my performance downstream,
 this routing function, especially also making this different during inference, you know, any
 any number of things, doing a Monte Carlo tree search at inference time to be as accurate as
 possible, kind of like, like AlphaGo or something. Do you have an idea on what the power of the
 routing function in these sparse models is? And how does it work currently? Like, what's the most,
 the latest and greatest? And how good is it? Yeah, so this is a really good question, actually,
 and something we've actually spent a lot of time about. So I would say actually, in this project,
 probably the thing I maybe spent the most time with is trying out different routing algorithms
 and routing parameterizations. But we ended up kind of going with the default thing, which I also
 think says something a little bit about the results of it. Yeah, so I would say my intuition is that
 the model actually works surprisingly well with a lot of different ways you can route
 the tokens. So like, you know, we tried a lot of other routing algorithms, we tried making like
 the routing network larger, we tried like, you know, some fancier ways of actually figuring out
 where you should send the token to, we tried, you know, using additional information of like,
 oh, when you're routing this current representation, you have access to whether or not like it was
 routed, or like where it was routed before in previous layers, using like word embedding
 information too. But yeah, I think overall, it seemed to be, you know, kind of insensitive,
 we actually did find like one or two methods that improve things, but they can only be used
 in certain situations. So it was a bit trickier to just like replace everything. The current routing
 algorithm we're using is basically what the original one was doing, I think in Shazir et al
 in 2017, when these kind of things were like really introduced into the LSTM language models.
 And I think, you know, our newer work, and then also Glam as well, we're using these kind of
 routing algorithms too. Yeah, and also like one kind of like detail here, it's like, so right now,
 we're sort of splitting out this little box, and we're like, oh, this is the router. It's not
 really an accurate characterization. It's like, yes, okay, you're mapping some vector into a
 vector that has like the same like length as number of experts. But if you just don't update
 that matrix, it still works fine, right? Because now just the represent like the weight matrices
 below you are just sort of adapting and just piping whatever activation they need, right?
 If you freeze the great if you stop a gradient through that, then it's like catastrophically bad.
 But yeah, I mean, I've also sort of been surprised by the relative insensitivity
 to the routing algorithm. Like we've seen like, you know, maybe some small boosts here and there,
 but it hasn't been super significant. I think you probably have a better sort of like a bigger
 significance by actually just sort of fundamentally changing like the architecture. Like maybe there's
 like some wildly different approach for sort of sparse models that we're not considering, maybe
 we're in some sort of like local men. And like these small tweaks on like, oh, okay, precisely,
 how are we doing this? Maybe doesn't matter as much. And DeepMinds also explored some other kind
 of interesting routing algorithms, like you sort of alluded to fixed routing algorithms, where it's
 just like, you're not even learning. They've also tried RL based routing algorithms. And I think it
 had like actually similar scaling properties. So again, kind of corroborating what Barrett is
 saying, it's just like, a lot of these things when we're kind of doing this, like per token routing,
 haven't really moved the needle substantially. That's been our our luck.
 Yeah, and I think another important trend actually, is that we when we were experimenting with a lot
 of these different routing algorithms, we actually found that they did help models. And maybe when
 you had like a 1 billion parameter dense modelish size, but then like, as we scaled up the models,
 like actually a lot of the time, sometimes the differences would just like wash away,
 as well. So it's kind of this interesting effect of when more scale is increased,
 like it maybe becomes a little bit less insensitive to some of these decisions.
 Yeah, I was I was Yeah, I can totally see that that essentially that the rest of the network
 adjusts, especially if everything is trainable. What I would be excited about maybe is is to
 somehow at inference time doing something smarter than because at training time, I can adjust to
 everything right, but at inference time, maybe there's something that I could do, especially
 with regards to, you know, domain shift, domain adaptation, anything like this, where I could,
 I could tweak routing in some way, but I guess that's also, also up for for future work.
 Okay. So there's a little bit of this not tweaking the routing algorithm, but tweaking the capacity
 factor hyper parameter I mentioned a while ago. So that's this is basically the parameter that's
 going to dictate how many tokens are being dropped. And one cool thing you can do is you can have some
 capacity factor during training. But then at eval time, depending on if you want to use more or less
 compute, you can be either dropping more or less tokens, and either kind of, you know, increase or
 decrease the performance, which is pretty cool. And the model is actually pretty robust to having
 that train from training evaluation time. So that's actually kind of like a good lever for like,
 you know, depending on if you want to use more or less compute during evaluation.
 I think we have we have a pretty good overview. Now I want to get a little bit into just the future,
 the future prospects, maybe also of this we already talked about, and with pathways, we could have
 heterogeneous things, could this be pushed to some sort of limit? Whenever I see a distributed
 system, you know, I immediately think distributed maybe not even in a data center, but across
 users across networks is their applications to maybe, what was it called federated, some kind of
 some kind of federated computing, some kind of federated learning where I could somehow contribute
 with my maybe confidential data, but I could still contribute to a whole compute process is there,
 I'm gonna say the the B word, is there an application for blockchain distribution,
 something like this? Like, have you do you think about sort of the higher degrees of distribution
 here? Do you want me to go for it? Yeah, go for it. I mean, yeah, so I mean, yes, me personally,
 I haven't spent a ton of time thinking about this. But I do think it's like very interesting. And
 yeah, there definitely seems to be a lot of really, you know, open problems around this,
 especially given the growing amount of like fragmented compute, fragmented devices, like
 there's so much computer on here, like, you know, how can you effectively utilize all of this,
 utilize different, you know, data and stuff, I think it's like a super cool and I think it
 was going to require a lot of really interesting research, because right now the way we're currently
 training these models is it's all like synchronized lockstep typically, right, you're doing like,
 oh, like after each batch, you do these gradients, you send the gradients around and everything. But
 like, I think actually, maybe the future of these models, when you're really, you know,
 allowing them to be distributed across very different types of computing, everything might
 actually now introduce like asynchronous training as kind of like the new paradigm. So I think
 that's like a really exciting space. But yeah, I haven't spent too much time thinking about it
 personally. Yeah, and I think like, as it pertains to say like blockchain or something, like, I think
 one problem with these expert models as designed in this way, are these all to all communications.
 So over this sort of like, you know, decentralized, like peer to peer network, where it's like, you
 know, nodes are like, you know, really far apart, inconsistent sort of bandwidth and stuff.
 That could be really tough if sort of your experts were sort of distributed among like many different
 nodes in this sort of like unreliable network where nodes are kind of coming and going. Like
 right now, all our systems are in this sort of like very constrained fault intolerant area where
 it's like, oh, all highly internet work chips that are highly reliable. And then so like blockchain
 would just have like a whole different set of like kind of problems that you'd have to sort of
 address like unreliability and you know, some of these other areas, not to say I think you just
 like require some like additional kind of research, like just sort of adopting the model as is, I think
 would pretty poorly map on that kind of computing infrastructure. But I think there's something there
 that could be done. Is there work on because I see these works mostly here in NLP yet transformers
 kind of taking over the rest of the world. Is there work on how these experts, sparse expert
 transformers behave in vision in reinforcement learning, speech, whatever? Yeah, yeah, great
 question. So absolutely, actually, there's been some really good work applying these models to
 like VIP based, like image classification and stuff. And there, it's actually really nice,
 because then you can leverage all of the, you know, niceties around like people figuring out
 how to get these working really well and transformers and kind of, you know, nicely map it over as well.
 I've, yeah, there's also been some good work using these in speech as well. Liam, any other
 things to add on top of that? Some, I used to do reinforcement learning more full time, and some
 colleagues kind of reached out about doing like sparse expert models for RL. I haven't seen, I'm
 not familiar with some work. But, you know, that might be sort of like another interesting avenue,
 but like for sure. So language, vision, speech. I don't know if there's been any videos,
 any video work yet. Yeah, but like high data, a lot of throughput, those would be like, you know,
 really good areas. So I think video would be also really promising. Yeah, I really like also the,
 I feel like it feels very natural in these high dimensionality spaces that you really might want
 different parameters to be applied, like when you have a video, like one, I think you don't want to
 be applying the same amount of compute to every frame. But then on top of that, I could see like,
 actually, you really want to have different parameters applying to different, you know,
 things going on in the video, because it's just gonna be like wildly different stuff happening.
 So yeah, I think I'm very excited about these models for video as well.
 Do you imagine that these models will just, essentially right now they're competition to
 dense models. They are competing, you're tracking Pareto frontiers, how much compute, how well are
 they doing, tackling very much the same tasks. Do you think this will go on? Like, do you think
 these models might overtake dense models if we figure out how to handle them correctly?
 Or is it more like there's a killer app for each one of them?
 Yeah, I think in, oh, do you want to go ahead, then? Yeah, I mean, I honestly think that the future
 is going to be adaptive. Like, I don't think there's any way that like in 10 years, our models are
 treating all examples coming in with like the same parameters over and over again, and the same amount
 of compute. It may not be this precise sort of like sparsity regime, or may not be the precise
 sort of adaptive computation, kind of like paradigms that have been put forth. But I view this sort of
 kind of work of like sparsity adaptive computation, as kind of like inevitable, like, I don't think
 it's going to be considered like competition, it's just going to be sort of like integrated into a
 lot of like leading models. That's, that's my expectation. I'd be really shocked in like 10
 years, we're training like a 100 trillion parameter dense model. And it's just kind of doing the same
 thing, like, over and over again, for no matter what comes in, just seems really strange to me.
 What's the future for your particular research? Like, where do you see, where do you see yourself
 going in the next, maybe not the next paper that you haven't published yet, but maybe a bit broader
 time scale? Like, what excites you? And what are your next plans here?
 Yeah, great question. I mean, I think the thing that really excites me is like what we were kind
 of talking about earlier of each input getting a different amount of compute applied. Like, I think
 right now, the models are working well for each input getting different parameters. And I think,
 you know, coupling this with like adaptive amounts of computation is like, I think,
 really where I want to be spending time thinking about in the next, you know, upcoming years.
 Is there? Yeah, I don't know, is you have something like Ponder, there's PonderNet,
 and so on, there's these recursive architectures, or recurrent architectures that, that sort of
 decide themselves when to when to exit. Would that be one thing? Or do you simply imagine that each
 expert is kind of one is the buff expert, and one is the lean expert, and then the routing function
 essentially takes care of the different amount of compute? Yeah, I don't know. This is a great
 question. I think, I don't know, I can see either approach potentially working, or maybe you
 actually want combinations or potentially something completely new. Yeah, it feels like the space is
 still, you know, very exciting. And there's like a lot of really interesting different verticals
 being pushed. So the space still feels like, you know, pretty young to me.
 Okay, last question from my side, what's the connection of this to something like Capsules?
 I don't know if you've ever thought about the the connection there. But with Capsules, I always
 think this is these abstract, very abstract things, very high level ideas flying around. And you here
 have something like very practical, you know, very on the metal thing. Yeah, there seems to be quite
 some commonalities. Is there is that something that ever came up to you? Or, or is that something
 that ever came up to you or? In the two years of doing sparsity research, this is literally the
 first time. I actually should be going back to that work. I feel like capsules like yeah, had a
 lot of like really interesting conceptions. But maybe like you're kind of alluding to it didn't
 like map super well to the metal. So maybe that sort of like hindered it's like it's use, whereas
 this is just like highly motivated from like an engineering perspective. We've had like some
 questions like, oh, what is like the neuroscientific kind of motivation of our work? And it's like,
 it's really engineering kind of driven. So it's like, okay, what what will be fast on our existing
 hardware? But yeah, I will revisit capsules and kind of see like, oh, okay, how could we actually
 map this a little bit better to the hardware? And like, you know, I think that could be like,
 you know, an interesting source of ideas. Is there any any last thing you want to get out to viewers
 that they should take away from this work? Any any way that a regular person can get into this type
 of research? Anything like this? Yes, a great question. So actually, one thing we tried to show
 in our switch transformer work is that these models work pretty well, even if you only have
 two experts. So I definitely I don't want people to think that, you know, you're really a supercomputer
 to run the models or to, you know, get benefits from having experts, even having I think, as little
 as two experts and running models could lead to developing really interesting research ideas,
 improving the performance and everything like that. So yeah, I definitely hope that, you know,
 more people can continue to experiment and push forward these models. Yeah, and then I would say,
 like, another interesting trend that I've been following is sort of in parallel to sparsity in
 these like, you know, really large models is the idea of like, well, what if we just sort of like,
 have the model sort of offload and like, sort of do lookups or, you know, look at documents and
 retrieval type methods. I think this is sort of like a very interesting area. And I'd love to see
 like, kind of head to head comparisons of like, okay, do we want to try to encapsulate the knowledge
 into parameters? Or do we want to just like, keep it sort of like, you know, parametric,
 non-parametric type thing? And we keep the information kind of written in docs or like,
 what does the interplay look like? I think that's sort of like another really interesting avenue,
 like, kind of comparing these things. Awesome. Yeah, it sounds really cool. I'm excited to
 to see what the future of these models bring. Yeah, Barrett and William, thank you so much
 for being here. This was a lot of fun. I hope to see you again soon. Yeah, cool. Thanks for having
 us. Yeah, thanks for having us.