GPT-NeoX-20B - Open-Source huge language model by EleutherAI (Interview w/ co-founder Connor Leahy)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper leahy eleuther eleutherai eleuther ai connor leahy coreweave gooseai goose ai gpt neo gpt-neo gpt-neox gpt-neox-20b gpt-j open source huggingface transformer transformer models gpt-3 open source gpt-3 download gpt-neox gpu cluster large language model large language models machine learning tutorial
#eleuther #gptneo #gptj

EleutherAI announces GPT-NeoX-20B, a 20 billion parameter open-source language model, inspired by GPT-3. Connor joins me to discuss the process of training, how the group got their hands on the necessary hardware, what the new model can do, and how anyone can try it out!

OUTLINE:
0:00 - Intro
1:00 - Start of interview
2:00 - How did you get all the hardware?
3:50 - What's the scale of this model?
6:00 - A look into the experimental results
11:15 - Why are there GPT-Neo, GPT-J, and GPT-NeoX?
14:15 - How difficult is training these big models?
17:00 - Try out the model on GooseAI
19:00 - Final thoughts

Read the announcement: https://blog.eleuther.ai/announcing-20b/
Try out the model: https://goose.ai/
Check out EleutherAI: https://www.eleuther.ai/
Read the code: https://github.com/EleutherAI/gpt-neox
Hardware sponsor: https://www.coreweave.com/

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Big announcement by a Luther AI releasing GPT Neo X 20 B. This is a 20 billion parameter large language model, and it will be publicly released in about a week from now. So less than a week from when you're seeing this. We have a blog post right now. So there will also be a paper coming up. The blog post details a little bit about the effort, a little bit about the model and releases some results on language modeling tasks and on factual knowledge tasks, where the model compares pretty good, pretty well against comparable baselines, not as good as something like GPT 3, which of course is 10 times larger, but it holds up quite well. And now I'm happy to welcome Connor Leahy, who is one of the founding members of a Luther AI and worked on GPT Neo X 20 B over the last months and even years, I guess. And we'll see what he has to say about it. Cool. Hey everyone. Today I have with me here Connor Leahy, who is one of the team members, founding members of the Luther AI and creators of GPT Neo 20, GPT Neo X 20 B model. Connor welcome. Thanks for having me on the show. It's really cool. I saw the announcement and let's, this is a big release, right? Yeah, so this whole thing was definitely like a year in the making overall. So we first started at CRC working on larger model like this with CoreWeave around, yeah, about a year ago. It's like probably like last February, maybe March, we had like starting time serious discussions. The chip shortage hit us. That was like a big problem to building the actual cluster and stuff. And just write the code and whatever. And yeah, finally we got to training about three months ago and yeah, got the model done like in the last couple of weeks and now pushed for release. So the cluster, you built a cluster for this model. It's not like there was one available, but you actually had to get hardware and so on. It's pretty cool. Like how does that work together with a hardware sponsor like CoreWeave? So CoreWeave have been really great to us. This wouldn't have been possible without them. Basically after we released the pile about a year ago and we kind of first had some variety of whatever, CoreWeave either December or January, I don't exactly remember when we first approached us, but they kind of first approached us and they're like, hey, let's do this. Like, you know, we want to get into large model training for our customers anyways. And we would like you guys like test our hardware to like help us find the right configurations of hardware. It was kind of like a back and forth kind of like, you know, we give them, you know, free testing, free advice, free consulting and in return, we get to use their cluster to build big models and release them. So like there was no financial exchange either way. It was just, you know, both helping each other. And you said, sorry, you said you delayed the release of the model, the weights for seven days due to your sponsors. Like what's that? Like why seven days? They asked for an exclusivity period so people would try it. Okay. That's basically it. So it's kind of the initial press bomb boost leads them. I mean, I tried it so it worked. Yeah. So, you know, we thought this was a very reasonable thing that we think doesn't like, isn't like a big compromise on our values or anything here. You know, we, our paper isn't finished yet anyway, so we probably would have delayed it anyways because we have finished writing our paper, which we want to release at the same time as we release the model. So this cost us basically nothing. It's good marketing for our friends. Everyone wins. Excellent. Give us a bit of this, like just the dimensions of the model right here. 20B is like, we've heard, like we're accustomed almost to this billion parameter models. What is it like scale of hardware, scale of just stuff that goes into it? What is it like? So the 20B model was trained on 96 A100s, all interconnected with SBX for, you know, NV switch interconnect and HDR InfiniBand. So this is all super high end data center quality hardware. As one of the things we learned while building the cluster and why we had built an actual cluster is at first, you know, Coreweave has like a ridiculous number of GPUs. They're like one of the biggest crypto miners and they, you know, provide like GPUs for like lots of like other services and whatnot. And so they have like thousands and thousands and thousands of GPUs. Unfortunately, the kind of GPUs you might use for crypto mining or first cloud gaming or for something like this, or usually single, you know, like single PCIe type GPUs. And those will not work for these large kinds of models where the bottleneck is really the communication between the individual chips. So you need this really low latency InfiniBand, you know, GPU to GPU direct interconnects and stuff if you want to have any hope of, you know, training these things. So you know, we tried like a bunch of like demo nodes that like didn't have NV switch or it didn't have InfiniBand or whatever. We kind of really worked our way up. And ultimately really this is the only thing that was possible and that's why we had to like kind of build it this way. So it was trained for three months on 96 A100s, which is quite a lot of, quite a lot of compute. And now the final model, if you want to use it for inference, it should run fine on any card, any GPU with about 48 gigabytes of memory or so. So it runs on an A6000 or an A40. Cool. Excellent. So the model will get into a little bit of the results right here. There's not too much yet. There's a press release. Your paper is going to come out. The model, as we said, are going to come out in about a week or so from time where we record this, but you have released some of the results. Can you give us maybe like a summary of the results, maybe something that was surprising to you or especially noteworthy? Yeah. So there's definitely a few interesting things that happened during the training and also with the eval results. So one funny thing that happened is during the training, our evals were really bad and we were kind of disappointed. But it turns out we actually had a bug in our code in like one of the operations, the defuse softmax. The way it was implemented caused it to give you bad results if you don't use the full context length for some reason. So the training was actually totally fine. And once you fix that bug, all of our benchmark jumped by like three or four percent. So that was nice. So the way the results currently look is the way I would describe it is it's a little less good at like natural language than maybe you would expect of a model of this size, but it is like a good bit better at like knowledge. This makes sense given the amount of the kind of data we've trained on. We train a lot of code. We trained on a lot of scientific papers, medical papers. So one of the things we did different in this model is we actually use a different tokenizer. So that's why comparing loss doesn't make sense to compare like complexity or loss to the other musts why we show like these accuracy numbers. So we use a tokenizer that we trained on the pile. And also we add like a bunch of like custom tokens or like multiple white space to like make code more efficient. So we tried like a bunch of different things, which in retrospect, we should have tried everything at once for the big model. We probably should have done more ablations before we started. If we have one piece of advice to people building big models, do ablations, do hyperparameter sweeps on small models. Really, really do that. That's really, really important. So yeah, so as a final result, I'm generally pretty happy. You know, it's not GPT-3 level. Of course not, because you know, DaVinci is a huge ass model and a really very, very well designed model. It compares pretty favorably. I think in most tasks, it's not, it doesn't knock anything really the park. I would say it's pretty good. It has a lot of very good knowledge, very good scientific knowledge. I haven't tried it yet very extensively myself to give you like a subjective impression of how it works. And one thing worth mentioning is the Hella swag results, which are just weird. We don't really know why they are so low. Like the Hella swag results specifically are like much lower than we would have expected them to be. We do not have an explanation for why that is. Okay. Short interjection. Connor actually told me later that they've mixed up two of these numbers, which means that Hella swag actually performs quite well. Yet it is the WSC that is not really explained why it's so bad. They suspect that it's the data set because JPT-J was already kind of bad on that model. But we don't know. Yet to be seen. Well, it seems that on the what we call standard language modeling tasks, it kind of holds itself, you know, holds par with let's say Fairsec or so is a bit behind DaVinci. And then on the factual knowledge tasks, it is quite a bit better than something like Fairsec, right? Yeah. Is that a function of... Because there is, I don't know, do you know Fairsec, what kind of data it was trained on? I don't know off the top of my head. Okay. Because there might be like a trade-off between, you know, model size may be responsible for some things and then data size or quality or nature might be responsible for another thing. It's pretty cool. Yeah. So I expect this to probably be down to the data. So because yeah, just the way the pile is built up and like because we also have a tokenizer specialized for the pile. So like the original GPT-2 tokenizer. So honestly, no one knows what tokenizers actually affect. Like no one has done any good studies on what different tokenizers do, whether large or small vocabularies are useful, whether you want, whether having words in your dictionary is good or bad. Like no one knows. This is all guessing basically. And so like, for example, our tokenizer has like, you know, really long medical terms as single tokens in it, but you know, sometimes lacks like some common, you know, words you might see in a book or something in its tokenizer, unlike other models. So I'm not too surprised that our model does pretty good on scientific things, which is generally, I think something we're interested in. I'm pretty sure if you would fine tune it, you would probably get really good results for other tasks as well. So like, as I was, you know, it's always important to caveat that this is, you know, an untuned model. This is a generally trained model. It can still be fine tuned. And yeah, we'd also don't know the best sampling parameters or whatever yet. So I'm sure people get a lot more performance. Same thing was happening with GPT-J when it first came out. When GPT-J first came out, it was horrible. Like every time you used it for anything, it was just awful. And then we turn, then for some reason, it's like, GPT-3 is pretty decent if you have it at temperature one, it's like not that bad. But for some reason, GPT-J just hates that. And you have to turn down temperature to like 0.8. Otherwise it's just awful. I can't explain why. It's just models have personality. And so there is this difference, right? There's GPT-J, which I understand is a JAX implementation. GPT-Neo-X has like a different code base. And the X is also an iteration on GPT-Neo, which was sort of your first project. Can you explain us a little bit, are these different people working on the different things? Like, why isn't there a GPT-J20B? So what's the reasoning behind sort of building these models, choosing the code bases, choosing what technologies to use? So it's mostly all by necessity. So we started with GPT-Neo when we only had access to TPUs from the Tenant Software Research Cloud as our sole compute. Neo is an incredibly cursed code base and should never be used by anyone. So Neo is fully deprecated, do not use Neo. We do not support Neo. We do not, don't even look at it. J is a offshoot in the sense, so yes, it is written completely in JAX, but it's done basically exclusively by Ben Wang. He basically just did that by himself, absolute mad lad. So it's kind of like an offshoot of the Aluthe AI project. So it's like a different type, different people worked on that than worked on Neo. The reason there is no J20B is that MTJ, so the actual code used to train 6B in my, if I'm remembering correctly, lack certain kinds of parallelisms that you would want for this large amount. You can do it, like we've tested it. It does kind of work, but it's pretty slow and we just can't reliably get enough TPUs to actually make that happen. So like, you know, we can get, you know, we've got like with 6B, you know, we just kind of just enough TPUs. I think it was 256 for like three weeks or so. And, you know, that took its time and it's very dependent on how much TPUs Google is currently using internally, whether we get access to some, because they're all preemptible. So we moved to Neox, which is written in PyTorch because we got GPUs, which is much nicer than TPUs. So yeah, that's basically the whole reason for that. So the people that worked on Neox are basically kind of the same people who worked on Neo. So big shout out to particular to Sid Black, who is like, you know, the figurehead for most of the Neo projects. Also, of course, too many people to name, but there's a lot of other people who have also contributed a lot. It's pretty cool to see that like different technologies matter because people are always like, well, you prefer TensorFlow or PyTorch or JAX and people are like, you know, whatever you want, like whatever fits. But as soon as you get to like these frontiers of engineering, it actually matters kind of. I mean, you could probably, as you said, implement anything in anything, but there the differences between can I do parallelism, can I do this or that, how easily can I do it? It's cool to see that there's still kind of a distinction between stuff and it's not just all like the same. My question is a bit, as you train these big model, you said ablations on small models to know your hyperparameters, how much handholding is required for the big models? Like how often do you have to like stop training, change something and then continue from where you stopped or this does not happen at all? Do you just restart and hope for better luck with some other parameters? So with 20b, we didn't have any like terrible problems, like things diverging and stuff like that. We of course did a lot of testing with hyperparameters, whatever, but honestly we could have done much more. So like large model training is very much alchemy, like you think ML is alchemy, this is the alchemy of the alchemy. Like it is very much secret recipes of like, for example, knowing that you set the Adam beta two parameter to 0.95 instead of 99 is really important. Like if you don't set it to 95, if you set it to 99, which is the default, you can't train large models, like it's like way more unstable. Come on, that's common knowledge. Oh yeah, common knowledge. Everyone would know these things. So yeah, it's just like, and like there's like so much of it is like folklore too. Like I remember someone asked someone at OpenAI like why do they use weight decay? And the answer was because Alec Redford said it helps. Like that's the whole reasoning why people use weight decay is because Alec Redford said it helps. Isn't there also like a difference between, I believe, isn't there a difference between the Adam parameters in the different frameworks, like the default parameters? Yeah, I think that is true. I don't know if it was off my head, but yeah, so like there's a lot of like little details like that that don't matter as much as smaller networks, but can really matter in large networks. So 20b, I think it's kind of like on the frontier of models that are still trainable in reasonable circumstances. So for example, the big science project from Hugging Face has been having an absolute hell of a time trying to train 100 billion parameter model and it just keeps diverging and then they roll it back or try something else and it diverges and they roll it back. We didn't have to do that with 20b. 20b was actually pretty well behaved, all things considered, once we had a set of parameters down and a pretty decent data set. Also very important, data set really matters. Like it really, really matters. Even the pile is like, we could do better now in retrospect. We're seeing like there's a lot of things like dedupeing and stuff that we could have done that we think would improve it quite a lot. So I remember, for example, the big science project once had those like huge divergence that like keep happening. And then they looked into the data set and they found that it was like 500,000 backslashes just consecutive that it was turning on. I mean, you got to see it, right? If you see, if you're gonna, yeah, it's better than 4chan, I guess. So people can try out this model. If they go to Goose AI, you can make an account and you can play around with it. A little bit. It's, it is the default model currently right here. I tried Hello and it did give me some code. Yeah, it gives me some code again. Do you, you said you haven't played around with it much, but is like what kind of stuff would you expect to work nicely? Anything? Do I have to set now the temperature to point A? I have no idea. So like I'm just saying like that's how it was with Jay. I don't know need to access personality. So I expect people to still find better, better parameters. Also like the playground, Goose AI is brand new. So I'm sure they're gonna add like more features like repetition penalty and stuff, which helps. So what I would expect New Ash to be a best at is code and like scientific tasks. Like, you know, so like, for example, I used to know a doctor who used like Jay and our Neo models to give him ideas for new research topics. Yeah. He would like prompt like, you know, I, you are a brilliant medical epidemiologist working in the field of XYZ and you are going to study and then it sometimes came up with really interesting experiments. I know that's like a common use case or whatever, but I would expect that to work. I'm sure it's fine at like, you know, you know, story generation and stuff like that. I would expect that like fine tuning it on more of those texts will probably make it a lot better. But yeah, it's knowledge should be pretty good. It should be pretty decent in coding, not as good as Codex or God forbid Alpha code, of course, but I would expect it to be pretty decent at all of these tasks. And this is still, this is still language modeling. So this is still like a likelihood next token prediction. This isn't any contrastive training or anything like this. Yep. Yep. This is just plain GPT-3 type training. Nice. Cool. Is there anything else you want to shout out about this model, people, code, anything? Well, I guess I just wanted to say, you know, thanks to the Lutri people also like to shout out maybe Anlanton and Aaron, who is their CEO, who has been very instrumental, including some of his employees have been really instrumental in helping with this. So this wasn't just Alutri AI, we also got a lot of help from them and some of the cluster stuff. And as you can see, they're also a partner on the Goose AI project. So we're very thankful for their help. It's been quite the ride. It's been good fun. We don't intend to stop here if you're, if you're interested in Alutri AI in the kind of work we do, or if you're an academic or research that wants to work on this kind of model, we'd love to hear from you. Check out our Discord. Love to hear from you. Connor, thank you very much for being with us.
 Big announcement by a Luther AI releasing GPT Neo X 20 B. This is a 20 billion parameter
 large language model, and it will be publicly released in about a week from now.
 So less than a week from when you're seeing this.
 We have a blog post right now.
 So there will also be a paper coming up.
 The blog post details a little bit about the effort, a little bit about the model and releases
 some results on language modeling tasks and on factual knowledge tasks, where the model
 compares pretty good, pretty well against comparable baselines, not as good as something
 like GPT 3, which of course is 10 times larger, but it holds up quite well.
 And now I'm happy to welcome Connor Leahy, who is one of the founding members of a Luther
 AI and worked on GPT Neo X 20 B over the last months and even years, I guess.
 And we'll see what he has to say about it.
 Cool.
 Hey everyone.
 Today I have with me here Connor Leahy, who is one of the team members, founding members
 of the Luther AI and creators of GPT Neo 20, GPT Neo X 20 B model.
 Connor welcome.
 Thanks for having me on the show.
 It's really cool.
 I saw the announcement and let's, this is a big release, right?
 Yeah, so this whole thing was definitely like a year in the making overall.
 So we first started at CRC working on larger model like this with CoreWeave around, yeah,
 about a year ago.
 It's like probably like last February, maybe March, we had like starting time serious discussions.
 The chip shortage hit us.
 That was like a big problem to building the actual cluster and stuff.
 And just write the code and whatever.
 And yeah, finally we got to training about three months ago and yeah, got the model done
 like in the last couple of weeks and now pushed for release.
 So the cluster, you built a cluster for this model.
 It's not like there was one available, but you actually had to get hardware and so on.
 It's pretty cool.
 Like how does that work together with a hardware sponsor like CoreWeave?
 So CoreWeave have been really great to us.
 This wouldn't have been possible without them.
 Basically after we released the pile about a year ago and we kind of first had some variety
 of whatever, CoreWeave either December or January, I don't exactly remember when we
 first approached us, but they kind of first approached us and they're like, hey, let's
 do this.
 Like, you know, we want to get into large model training for our customers anyways.
 And we would like you guys like test our hardware to like help us find the right configurations
 of hardware.
 It was kind of like a back and forth kind of like, you know, we give them, you know,
 free testing, free advice, free consulting and in return, we get to use their cluster
 to build big models and release them.
 So like there was no financial exchange either way.
 It was just, you know, both helping each other.
 And you said, sorry, you said you delayed the release of the model, the weights for
 seven days due to your sponsors.
 Like what's that?
 Like why seven days?
 They asked for an exclusivity period so people would try it.
 Okay.
 That's basically it.
 So it's kind of the initial press bomb boost leads them.
 I mean, I tried it so it worked.
 Yeah.
 So, you know, we thought this was a very reasonable thing that we think doesn't like, isn't like
 a big compromise on our values or anything here.
 You know, we, our paper isn't finished yet anyway, so we probably would have delayed
 it anyways because we have finished writing our paper, which we want to release at the
 same time as we release the model.
 So this cost us basically nothing.
 It's good marketing for our friends.
 Everyone wins.
 Excellent.
 Give us a bit of this, like just the dimensions of the model right here.
 20B is like, we've heard, like we're accustomed almost to this billion parameter models.
 What is it like scale of hardware, scale of just stuff that goes into it?
 What is it like?
 So the 20B model was trained on 96 A100s, all interconnected with SBX for, you know,
 NV switch interconnect and HDR InfiniBand.
 So this is all super high end data center quality hardware.
 As one of the things we learned while building the cluster and why we had built an actual
 cluster is at first, you know, Coreweave has like a ridiculous number of GPUs.
 They're like one of the biggest crypto miners and they, you know, provide like GPUs for
 like lots of like other services and whatnot.
 And so they have like thousands and thousands and thousands of GPUs.
 Unfortunately, the kind of GPUs you might use for crypto mining or first cloud gaming
 or for something like this, or usually single, you know, like single PCIe type GPUs.
 And those will not work for these large kinds of models where the bottleneck is really the
 communication between the individual chips.
 So you need this really low latency InfiniBand, you know, GPU to GPU direct interconnects
 and stuff if you want to have any hope of, you know, training these things.
 So you know, we tried like a bunch of like demo nodes that like didn't have NV switch
 or it didn't have InfiniBand or whatever.
 We kind of really worked our way up.
 And ultimately really this is the only thing that was possible and that's why we had to
 like kind of build it this way.
 So it was trained for three months on 96 A100s, which is quite a lot of, quite a lot of compute.
 And now the final model, if you want to use it for inference, it should run fine on any
 card, any GPU with about 48 gigabytes of memory or so.
 So it runs on an A6000 or an A40.
 Cool.
 Excellent.
 So the model will get into a little bit of the results right here.
 There's not too much yet.
 There's a press release.
 Your paper is going to come out.
 The model, as we said, are going to come out in about a week or so from time where we record
 this, but you have released some of the results.
 Can you give us maybe like a summary of the results, maybe something that was surprising
 to you or especially noteworthy?
 Yeah.
 So there's definitely a few interesting things that happened during the training and also
 with the eval results.
 So one funny thing that happened is during the training, our evals were really bad and
 we were kind of disappointed.
 But it turns out we actually had a bug in our code in like one of the operations, the
 defuse softmax.
 The way it was implemented caused it to give you bad results if you don't use the full
 context length for some reason.
 So the training was actually totally fine.
 And once you fix that bug, all of our benchmark jumped by like three or four percent.
 So that was nice.
 So the way the results currently look is the way I would describe it is it's a little less
 good at like natural language than maybe you would expect of a model of this size, but
 it is like a good bit better at like knowledge.
 This makes sense given the amount of the kind of data we've trained on.
 We train a lot of code.
 We trained on a lot of scientific papers, medical papers.
 So one of the things we did different in this model is we actually use a different tokenizer.
 So that's why comparing loss doesn't make sense to compare like complexity or loss to
 the other musts why we show like these accuracy numbers.
 So we use a tokenizer that we trained on the pile.
 And also we add like a bunch of like custom tokens or like multiple white space to like
 make code more efficient.
 So we tried like a bunch of different things, which in retrospect, we should have tried
 everything at once for the big model.
 We probably should have done more ablations before we started.
 If we have one piece of advice to people building big models, do ablations, do hyperparameter
 sweeps on small models.
 Really, really do that.
 That's really, really important.
 So yeah, so as a final result, I'm generally pretty happy.
 You know, it's not GPT-3 level.
 Of course not, because you know, DaVinci is a huge ass model and a really very, very well
 designed model.
 It compares pretty favorably.
 I think in most tasks, it's not, it doesn't knock anything really the park.
 I would say it's pretty good.
 It has a lot of very good knowledge, very good scientific knowledge.
 I haven't tried it yet very extensively myself to give you like a subjective impression of
 how it works.
 And one thing worth mentioning is the Hella swag results, which are just weird.
 We don't really know why they are so low.
 Like the Hella swag results specifically are like much lower than we would have expected
 them to be.
 We do not have an explanation for why that is.
 Okay.
 Short interjection.
 Connor actually told me later that they've mixed up two of these numbers, which means
 that Hella swag actually performs quite well.
 Yet it is the WSC that is not really explained why it's so bad.
 They suspect that it's the data set because JPT-J was already kind of bad on that model.
 But we don't know.
 Yet to be seen.
 Well, it seems that on the what we call standard language modeling tasks, it kind of holds
 itself, you know, holds par with let's say Fairsec or so is a bit behind DaVinci.
 And then on the factual knowledge tasks, it is quite a bit better than something like
 Fairsec, right?
 Yeah.
 Is that a function of...
 Because there is, I don't know, do you know Fairsec, what kind of data it was trained
 on?
 I don't know off the top of my head.
 Okay.
 Because there might be like a trade-off between, you know, model size may be responsible for
 some things and then data size or quality or nature might be responsible for another
 thing.
 It's pretty cool.
 Yeah.
 So I expect this to probably be down to the data.
 So because yeah, just the way the pile is built up and like because we also have a tokenizer
 specialized for the pile.
 So like the original GPT-2 tokenizer.
 So honestly, no one knows what tokenizers actually affect.
 Like no one has done any good studies on what different tokenizers do, whether large or
 small vocabularies are useful, whether you want, whether having words in your dictionary
 is good or bad.
 Like no one knows.
 This is all guessing basically.
 And so like, for example, our tokenizer has like, you know, really long medical terms
 as single tokens in it, but you know, sometimes lacks like some common, you know, words you
 might see in a book or something in its tokenizer, unlike other models.
 So I'm not too surprised that our model does pretty good on scientific things, which is
 generally, I think something we're interested in.
 I'm pretty sure if you would fine tune it, you would probably get really good results
 for other tasks as well.
 So like, as I was, you know, it's always important to caveat that this is, you know, an untuned
 model.
 This is a generally trained model.
 It can still be fine tuned.
 And yeah, we'd also don't know the best sampling parameters or whatever yet.
 So I'm sure people get a lot more performance.
 Same thing was happening with GPT-J when it first came out.
 When GPT-J first came out, it was horrible.
 Like every time you used it for anything, it was just awful.
 And then we turn, then for some reason, it's like, GPT-3 is pretty decent if you have it
 at temperature one, it's like not that bad.
 But for some reason, GPT-J just hates that.
 And you have to turn down temperature to like 0.8.
 Otherwise it's just awful.
 I can't explain why.
 It's just models have personality.
 And so there is this difference, right?
 There's GPT-J, which I understand is a JAX implementation.
 GPT-Neo-X has like a different code base.
 And the X is also an iteration on GPT-Neo, which was sort of your first project.
 Can you explain us a little bit, are these different people working on the different
 things?
 Like, why isn't there a GPT-J20B?
 So what's the reasoning behind sort of building these models, choosing the code bases, choosing
 what technologies to use?
 So it's mostly all by necessity.
 So we started with GPT-Neo when we only had access to TPUs from the Tenant Software Research
 Cloud as our sole compute.
 Neo is an incredibly cursed code base and should never be used by anyone.
 So Neo is fully deprecated, do not use Neo.
 We do not support Neo.
 We do not, don't even look at it.
 J is a offshoot in the sense, so yes, it is written completely in JAX, but it's done basically
 exclusively by Ben Wang.
 He basically just did that by himself, absolute mad lad.
 So it's kind of like an offshoot of the Aluthe AI project.
 So it's like a different type, different people worked on that than worked on Neo.
 The reason there is no J20B is that MTJ, so the actual code used to train 6B in my, if
 I'm remembering correctly, lack certain kinds of parallelisms that you would want for this
 large amount.
 You can do it, like we've tested it.
 It does kind of work, but it's pretty slow and we just can't reliably get enough TPUs
 to actually make that happen.
 So like, you know, we can get, you know, we've got like with 6B, you know, we just kind of
 just enough TPUs.
 I think it was 256 for like three weeks or so.
 And, you know, that took its time and it's very dependent on how much TPUs Google is currently
 using internally, whether we get access to some, because they're all preemptible.
 So we moved to Neox, which is written in PyTorch because we got GPUs, which is much nicer than
 TPUs.
 So yeah, that's basically the whole reason for that.
 So the people that worked on Neox are basically kind of the same people who worked on Neo.
 So big shout out to particular to Sid Black, who is like, you know, the figurehead for
 most of the Neo projects.
 Also, of course, too many people to name, but there's a lot of other people who have
 also contributed a lot.
 It's pretty cool to see that like different technologies matter because people are always
 like, well, you prefer TensorFlow or PyTorch or JAX and people are like, you know, whatever
 you want, like whatever fits.
 But as soon as you get to like these frontiers of engineering, it actually matters kind of.
 I mean, you could probably, as you said, implement anything in anything, but there the differences
 between can I do parallelism, can I do this or that, how easily can I do it?
 It's cool to see that there's still kind of a distinction between stuff and it's not just
 all like the same.
 My question is a bit, as you train these big model, you said ablations on small models
 to know your hyperparameters, how much handholding is required for the big models?
 Like how often do you have to like stop training, change something and then continue from where
 you stopped or this does not happen at all?
 Do you just restart and hope for better luck with some other parameters?
 So with 20b, we didn't have any like terrible problems, like things diverging and stuff
 like that.
 We of course did a lot of testing with hyperparameters, whatever, but honestly we could have done
 much more.
 So like large model training is very much alchemy, like you think ML is alchemy, this
 is the alchemy of the alchemy.
 Like it is very much secret recipes of like, for example, knowing that you set the Adam
 beta two parameter to 0.95 instead of 99 is really important.
 Like if you don't set it to 95, if you set it to 99, which is the default, you can't
 train large models, like it's like way more unstable.
 Come on, that's common knowledge.
 Oh yeah, common knowledge.
 Everyone would know these things.
 So yeah, it's just like, and like there's like so much of it is like folklore too.
 Like I remember someone asked someone at OpenAI like why do they use weight decay?
 And the answer was because Alec Redford said it helps.
 Like that's the whole reasoning why people use weight decay is because Alec Redford said
 it helps.
 Isn't there also like a difference between, I believe, isn't there a difference between
 the Adam parameters in the different frameworks, like the default parameters?
 Yeah, I think that is true.
 I don't know if it was off my head, but yeah, so like there's a lot of like little details
 like that that don't matter as much as smaller networks, but can really matter in large networks.
 So 20b, I think it's kind of like on the frontier of models that are still trainable in reasonable
 circumstances.
 So for example, the big science project from Hugging Face has been having an absolute hell
 of a time trying to train 100 billion parameter model and it just keeps diverging and then
 they roll it back or try something else and it diverges and they roll it back.
 We didn't have to do that with 20b.
 20b was actually pretty well behaved, all things considered, once we had a set of parameters
 down and a pretty decent data set.
 Also very important, data set really matters.
 Like it really, really matters.
 Even the pile is like, we could do better now in retrospect.
 We're seeing like there's a lot of things like dedupeing and stuff that we could have
 done that we think would improve it quite a lot.
 So I remember, for example, the big science project once had those like huge divergence
 that like keep happening.
 And then they looked into the data set and they found that it was like 500,000 backslashes
 just consecutive that it was turning on.
 I mean, you got to see it, right?
 If you see, if you're gonna, yeah, it's better than 4chan, I guess.
 So people can try out this model.
 If they go to Goose AI, you can make an account and you can play around with it.
 A little bit.
 It's, it is the default model currently right here.
 I tried Hello and it did give me some code.
 Yeah, it gives me some code again.
 Do you, you said you haven't played around with it much, but is like what kind of stuff
 would you expect to work nicely?
 Anything?
 Do I have to set now the temperature to point A?
 I have no idea.
 So like I'm just saying like that's how it was with Jay.
 I don't know need to access personality.
 So I expect people to still find better, better parameters.
 Also like the playground, Goose AI is brand new.
 So I'm sure they're gonna add like more features like repetition penalty and stuff, which helps.
 So what I would expect New Ash to be a best at is code and like scientific tasks.
 Like, you know, so like, for example, I used to know a doctor who used like Jay and our
 Neo models to give him ideas for new research topics.
 Yeah.
 He would like prompt like, you know, I, you are a brilliant medical epidemiologist working
 in the field of XYZ and you are going to study and then it sometimes came up with really
 interesting experiments.
 I know that's like a common use case or whatever, but I would expect that to work.
 I'm sure it's fine at like, you know, you know, story generation and stuff like that.
 I would expect that like fine tuning it on more of those texts will probably make it
 a lot better.
 But yeah, it's knowledge should be pretty good.
 It should be pretty decent in coding, not as good as Codex or God forbid Alpha code,
 of course, but I would expect it to be pretty decent at all of these tasks.
 And this is still, this is still language modeling.
 So this is still like a likelihood next token prediction.
 This isn't any contrastive training or anything like this.
 Yep.
 Yep.
 This is just plain GPT-3 type training.
 Nice.
 Cool.
 Is there anything else you want to shout out about this model, people, code, anything?
 Well, I guess I just wanted to say, you know, thanks to the Lutri people also like to shout
 out maybe Anlanton and Aaron, who is their CEO, who has been very instrumental, including
 some of his employees have been really instrumental in helping with this.
 So this wasn't just Alutri AI, we also got a lot of help from them and some of the cluster
 stuff.
 And as you can see, they're also a partner on the Goose AI project.
 So we're very thankful for their help.
 It's been quite the ride.
 It's been good fun.
 We don't intend to stop here if you're, if you're interested in Alutri AI in the kind
 of work we do, or if you're an academic or research that wants to work on this kind of
 model, we'd love to hear from you.
 Check out our Discord.
 Love to hear from you.
 Connor, thank you very much for being with us.