 Hey, fun fact. Guys, did you know that max sequence input length is actually not the same as the context window? Wiz, Leo, isn't that right? Yeah. Pretty true, Greg. Yeah, yeah, yeah, yeah, yeah. Fun fact, and we will dive into all of the details that we need to go long on context today. You guys ready to get into it? Let's go. Let's do it. Ready. Yep. All right. We'll have you guys back in just a sec. Today, we go long on context with the chief scientist from Gradient, Leo Pichelis, and of course, the Wiz. Today, we're talking about exactly how these context windows should be thought about, how we can make them longer, what this requires, and how it's happening in real time. And of course, once we have the long context window, how do we know that it's actually performing well? How can we evaluate the long context window performance? So we're going to set this off by providing some context. Then we're going to talk about what goes into actually creating an off-the-shelf model that then gets expanded with its context window. And we're going to talk about some of the popular frameworks for evaluating long context window LLMs. We're going to talk about some of the popular frameworks for evaluating long context window LLMs. So I want to sort of first provide a little background here, a little context. In April, of course, Meta released Lama 3. And what was so interesting is when they put out the blog, they said, well, we're not going to do the long context window for a little while. We're working on it. We've got a bigger model. We've got some longer context versions. And everybody said, man, it's only 8K input sequence length. I don't know. I'm really left wanting with this small amount that I can put into this thing. And the race was on in the industry. Gradient, friends of ours, were instantly off to the races, crushing it. Somehow, within just a few days, working maniacally over the weekend, we understand, released 160K context length version of the 8 billion parameter model before they then released a few days later a 1 million context length version of the 8B. And then they released a 262 context length version of the 70B before they just went straight buck wild on May the 4th with their 1 million context length version of 70B. They've actually released a 4 million length version of 8B now, and it's really just kind of hard to keep track of all the cool stuff that they're doing so i wanted to sort of get leo up on stage to tell you a little bit about what this has been like behind the scenes at gradient so leo i mean how did you guys pull this off before meta and so quickly what's it been like down in the trenches and like what should we expect to see from you guys next yeah yeah thanks um it's definitely been a a lot of work kind of like as you mentioned and you know a number of uh nights and weekends i think a a lot of it is being in the right place at the right time um we were coincidentally already working on some long context stuff before Llama 3 dropped. We were working with our compute partner, actually, Crusoe, and they had just spun up, I think it's a thousand GPU L40s cluster. And we were kind of like throwing the ball around, trying to figure out uh what's something uh really interesting that we could work on together and we settled on on long context right i think over the past few months like really just this year people been uh putting out long context models there was like the large world model uh maybe like a month or two ago up to like a million context uh google gemini has been talking about it too um and so we were coincidentally working on some stuff um and it was kind of the perfect storm too because it wasn't just us it was kind of like the whole open source community uh there's this great easy context github repo out there like again literally like released maybe like two weeks or three weeks before llama 3 and so it we were in the right place at the right time working on the right stuff heard about llama 3 dropping and we're like great let's let's see how it would work and it all kind of just fell together it wasn't much of a pivot then it was just sort of like an acceleration of some of you guys were already working on behind the scenes? Yeah, yeah, exactly. I mean, we were mostly working on like Lama 2 architectures. And so like there's a little bit of like a hope and pray that all of this would just kind of work for Lama 3 and like credit to Meta and their team, it did. It kind of worked flawlessly. Nice, nice. So, I mean, you know, we saw the 4 million input sequence length. What's next? What are you guys working on now? And I mean, how long are we going here exactly? Yeah. I mean, you could potentially take it as long as you want. But, you know, at a certain point, there's diminishing returns, right? Like 4 million context length. I think it's something like 1.25 tokens per word or something like that. But anyways, at like 4 million, you've got like the whole Harry Potter book series a number of times over. And you figure at the end of the day, Gradient serves enterprise customers, right? Like we build agent systems for enterprise company. And something like four or five million is pretty much kind of like where the demand that we've been seeing is. And so I think probably the pivot from here is making those models better. I think in that direction, there's tons of room to improve, both on kind of like evaluating and figuring out, well, okay, it can like read long context, but maybe you want it to like reason about long context. And then also just working on the applications, right? I think we're just barely scratching the surface of all the cool stuff we could do with it. Yeah, yeah, yeah. I like this idea of sort of enterprise and they have sort of a kind of context length that seems to be sort of this diminishing returns point. This might be an interesting place for us to come back to in our discussions today. And then, you know, this idea of exactly what better means, I think we'll cross that path as well. Thanks for giving us some initial context, Leo. We'll have you back in just a little bit to keep the discussion rolling. For now, what we're going to talk about is a little background. And it's important to understand sort of this idea in general of in-context learning is kind of fundamental. This came out with the GPT-3 paper called Language Models or Few Shot. And it's it's all in the figure caption right here. Larger models make increasingly efficient use of in context information in the previous GPT one and GPT two paradigms long time ago now, but they were really requiring a lot of this fine tuning. And what we saw with the in-context learning enabled GPT-3 is that if we just did zero shot, that is instruction only, one shot, meaning one example of the kind of thing you wanted to do into few shot, especially at large enough model sizes, we're getting very, very, very performant results across many tasks. And in fact, if you've watched our channel before, you've noticed that we oftentimes talk about RAG as fundamentally requiring this in-context learning piece. And, you know, it's a bit jargony to even say in-context learning. What we're really talking about is we're talking about using the prompts, right? We're talking about doing some prompt engineering, using the input to the LLM, giving clear and specific instructions. And of course, we can talk about providing a context during our prompting. We can say, hey, you're a specific role. You should act a specific way. You have a specific feeling. But more importantly, when we want to fact checkable things, we're oftentimes saying, hey, we actually want to worry about this retrieved context piece. Because, of course, as we move from prompt engineering into RAG, we're sort of optimizing the context. That's fundamentally what we're doing. And as we go from zero shot to one shot to shot, to many shot learning, we're just giving more and more examples. These are sort of task-specific things. But in RAG, we're kind of giving more reference material. In either case, we're sort of giving the LLM access to knowledge that it wasn't really trained on in the first place. And the way that this works in a retrieval augmented generation system is we ask a question, it gets converted to a vector representation, and we've stored all of our potentially relevant data locally in a database called a vector store. locally in a database called a vector store. And then we look for stuff that's similar to our question. The stuff that we find that's similar to our question, we put and inject directly into our prompt using a prompt template that says something like use the provided context to answer the user's query. You context to answer the user's query. You may not answer the user's query unless there's specific context. If you don't know the answer, say I don't know. And then what we return is we return natural language directly to this context. And this retrieval process is the R in RAG, but this augmentation that we're doing to the prompt, that's in context learning. And so the RA into the G that is finally the generation, that's really the whole process here to get our answer. Now, in terms of what we're talking about doing now is we're saying, well, what if we just had a question and then we just copy pasted the entire Harry Potter series a couple of times over, right? Then how about just like, bam, let's just shove it into the model and get our answer. This is kind of the long context approach. And you might say, well, why do people want to do this? And I mean, clearly it's a no brainer, right? Because it doesn't get easier than doing something like this. So let's bring our experts back up to the stage here to have a little discussion about the big question that we are constantly getting here, guys. Why do we even need RAG if we have long context windows? They're going to kill RAG, right? Let's start off with that. Leo, what are your thoughts on this? Yeah, I mean, I don't think RAG is dead if you get long context, right? But I do think it looks a little bit different. You know, like one of the things that RAG really depends on is you got to pull out the right information. And that's why, like, people are working on a lot of, like, re-rankers. Like, you pull out more information than you think you need and then you kind of, like, sift through it. That stuff is, like, pretty complicated. And like, isn't it great if you could just pull out more information than you need throw it all into the model and let it kind of like figure out what it uses and what it doesn't use um and i think like especially the part that like rag is like pretty it would be pretty hard to achieve with just rag is when the pieces of information are interrelated, right? It's like, one example is say, and it's kind of a silly example, but, you know, maybe in one document, you have people's birthdays, right? Like maybe Sally's birthday is May 22nd. And in another document, you have Sally went to the store and bought like apples and oranges. And you ask the rag, what does someone whose birthday is in May, like buy at the store? Because those two pieces are interrelated, I think it would take an LLM multiple queries, but if you had it all in the context, it'd be just like, well, apples and oranges. So I think like that kind of interrelated stuff is really where the long context shines. Yeah. Yeah, so this sort of long range interrelations and correlations, I mean, I think this is something you've thought quite a lot about all the questions we get from students was, and maybe you can kind of comment on this quote that I took directly from you here and to kind of color the discussion if we built a perfect retrieval system then we wouldn't need long context is this true or false in your opinion and talk a little bit about why you think so well i feel like because i said it i'm gonna have to say that it's true uh The idea, I think, is right. If we have perfect retrieval, then we always get the information that we need to answer the question correctly. So we don't really need long context. We don't necessarily need long context, assuming that the context we need to answer the question isn't super long, right? And for most questions, that's going to be reasonably true. But that perfect retrieval system is not a trivial problem. Let's like, you know, be a little silly about it. But it's a very, very difficult problem, in fact. And I think, you know, long context can help us fudge that perfect context by just including more information so that we're more likely to have the correct context when we're looking to do anything with that context. Okay. To qualify a bit. That's what it is. Yeah, yeah, yeah, yeah. I mean, like, so this is clearly a hot clearly a hot take. Now I want to bring in, Leo, you mentioned enterprise earlier and you mentioned this four to five million length. I mean, enterprises got to be asking themselves this question. How are you thinking about providing enterprise the best solution, you know, that sort of balances great retrieval with the right amount of context are you typically leveraging both of these tools uh in any given use case uh maybe just tell us a little bit about um something that uh you know maybe you can tell us about right no happy happy to to share uh i know people people might get mad at me at work afterwards, but happy to share whatever. Yeah, I think the one thing I was going to say about the perfect retrieval is I think you can think about long context as a way to get at perfect retrieval. I think that's kind of what you're getting at as well um you know it's like i always like to think about analogies and like for us like in the human world and you know i could ask um i don't know like like a like a researcher or like a lawyer or someone to find me the yeah maybe like lawyer right like ask them to find me like the best counter argument for this particular case and and yeah like what they would probably do is like read through like a whole ton of different like prior cases find out like the few examples that are really worthwhile and like compile them and and return them right and like that one piece of information they return might very well be pretty short context right you know lawyers get get paid bait by the hour it's probably you know worthwhile to a lot of folks to to have it be a concise summary but there's a ton of work that went into that right and and so maybe if you have the llm go through all those pages and pages of briefs to to end up with that perfect retrieval that's that's like one way to do it. You know, you were asking about like the cost considerations for enterprise and totally that's like something to think about because I think the trade-off that you're making by using a long context window is it is probably more costly than doing a RAG query. You know, it takes a certain number of GPUs. I think on eight GPUs with an 8B model, we're getting up to 500K or maybe 600K context. Anything past that, we got to get beefier GPUs, we've got to get more of them. And so, you know, for any kind of like where the rubber meets the road system, you got to think about ROI, right? So, you know, maybe you don't want to throw like the whole internet or like, you know, Harry Potter five times over into the context, maybe some of it you want in the rag and some of it you want in the long context. Okay. Okay. Okay. So long context then sort of in spirit is kind of moving towards this idea of a perfect retrieval system. Something like, you know, we were chatting the other day about, Wiz and I were chatting about kind of like whether using semantic chunking or you're using a hybrid retrieval mechanism or you're using re-ranking you're sort of trying to figure out what is most relevant in the context right and when we do the assessment with tools like rag assessment we're sort of saying we don't want very duplicative information we don't want uh you know stuff that's going to be sort of just copy paste of the sameness, but we also want sort of rank ordered relevant information. I mean, you know, we talk about the problem RAG is solving as sort of being fact checkable. Leo, do you think that's the same problem long context is solving or can we sort of state it a little bit differently? I mean, are we kind of moving in the same direction with these two tools? Yeah, I think it's definitely moving in the same direction. I mean, fact checkable is such an interesting term because if you think about like all of the data that a language model is trained on it's it's trained on facts right it's trained on information that's out there um but like you know hallucination the model maybe is is extrapolating or it's like connecting pieces of information that maybe it shouldn't be connecting right and and all of this kind of like learning is trying to trying to teach the model both about new facts and how it maybe like shouldn't connect pieces of information that it has. And so like fact checking, I think you can totally do with long context because the facts are in the context. And I think there's been plenty of research to show that kind of like you mentioned in context learning, way more sample efficient than trying to pre-train or fine tune the model, right? And so that can be kind of like your last step fact checking that is very sample efficient, I think similar to RAG. I think similar to RAG. I think one of the things that I like about putting more and more into the language model is this kind of principle of just getting out of the way of the model and letting it learn, right? Like there's something beautiful about the fact that you have a very sophisticated AI model. You feed it examples and information, and it is able to synthesize together how it retrieves information or how it ranks pieces of data versus necessarily needing to code up the algorithm by hand, right? Like let the model figure it out. So one of the things we did before is we sort of said rag versus fine tuning. And it's sort of like you need both to kind of get to this sort of production grade LM application. Like, is it similar here with rag and long context? Like, what's at the end of this tunnel? Is this an illusion of choice or is it real? You know, Wiz, give me your take on this. And then we'll go to Leo before we kind of wrap this rag versus long it's just so tough to grok this sort of like when do I use either exactly you know and I think it's the new hottest question out there yeah I mean I mean uh for me this question is is pretty easy. It's both. I think that there are times when we're going to be able to use, you know, RAG and long context to make sure that we're doing, you know, getting everything that we need or up to everything that we need, especially if we're thinking about kind of API-based solutions where we're not actually needing to host those long context models, right, especially if we're thinking about kind of API based solutions where we're not actually needing to host those long context models, right, so that the flexibility of the context window is almost explicitly free for us, right? We're only paying if we use it. And especially if we're, you know, we're thinking about RAG across very specific sets of documents that don't change much or don't change frequently that we want to do retrieval across, we can kind of exploit the fact that we can hold this long context in memory, right? So we don't actually have to do these repeated loads and unloads. So we're going to wind up with less of a latency hit. The idea is that at some point, you know, especially as we continue to discover more about, you know, efficient long context, we're going to be able to do both. And, you know, it's one of those things where if one is 2% better on one task and the other is, you know, 2% better on the other task, and there's not a serious cost to use both then we should probably we should probably just use both right like it'll it'll be better than uh than either individually yeah yeah okay because you can use caching with both you can use sort of the these these cost memory saving sort of things with both you can uh are they are they both sort of becoming just, you know, table stakes here, Leo? Is that is that sort of fair to say? How would you like to sort of wrap this for the audience here? Yeah, yeah. I mean, I think that's like actually very probably like like a pretty deep statement in the fact that they're actually not that dissimilar at the end of the day. And yes, definitely using both. And my suspicion is that as we, again, super at the frontier of all of this stuff, but as we go forward, I imagine that rag and long context will just be parts of a spectrum. And the one piece of evidence I have in my mind is, again, don't mean to sound like a broken record, but like thinking about like people and humans, like the way our minds work is there is like some piece of information that we have like packed up back there, like somewhere in our brains that we reference and we look up and there's some pieces of information that we have at the forefront. Right. Like I was reviewing all the work we did for long context before this talk. And that's like in my current long context, we do both. So I think it stands to reason that AIs are going to do both just the same. Yeah. Yeah. You're studying for the exam. That's the long context. Right. But like the sort of like TED talk nuggets that like entered your brain and solidified and crystallized maybe that's the rag yeah super exactly things stuff guys okay I'm gonna give a little more context here before the next discussion thank you so much guys we'll have you back in just a moment so I want to talk about the context window now, because this is sort of an important point that I think goes misunderstood and understated. When we are thinking about the problem that we're solving with long context, we're talking about the fact that the model has only learned correlations across sequences of length, whatever it was trained on. So the Harry Potter books keep coming up. It doesn't know things across the whole series necessarily because it's never seen a single input sequence of that length. And this is where we get into this sort of distinction between context window and input sequence length. Let's take a look at some of the latest models. Llama3 said, we trained the models, quote, "'trained the models on sequences of 8,000 tokens." Trained the model. They didn't say the context window was this length. Whereas GPT-4 Omni, they say the context window, 128K. The thing that I want to tell all of you today is that really there is no sort of in and out when it comes to context window. There is only context window. So I was looking through the GPT, I guess, forums, the OpenAI forums, and there was just a great sort of upvoted, you know, hey, what exactly is the difference in input? The context window includes all input, output, and control tokens. Now, this is very important because the context window is to be thought of as all the stuff that goes in and all the stuff that can be held that comes out. So this is kind of a difficult thing to conceptualize. And I want to bring our experts back up to the stage to talk a little bit about how they think about this context window. I think oftentimes, guys, we see that there's a sort of context window length that you can put into an LLM and everybody thinks like that's the end of the story here but actually it's more than that it's the input and output that need to be considered and so like how do you think about context window and balancing your understanding of input i.e. sort of max input sequence length with context window in general. Let's kick it off with you, Leo, here. Yeah. I think like on a like slightly more like technical side of things, the whole like you're absolutely right, right? Like the context window is everything, right? It's like the system prompt it's what you put in and it's what you get out uh the reason that's true is is exactly what makes like these AI models so powerful is that it is able in its memory for figuring out the next token or the next word reference everything that's been it's seen before right and so like when it when it gets to its output in order to to tell you the right thing uh it needs to know about everything that you've inputted and also you know you mentioned open ai like everything open ai put in the system prompt um and and so that's exactly why like technically speaking the the context window is is everything and like you know when it gets to the very last word the output it needs to know everything that it's written before and everything that you gave it and also the system prompt um yeah and so like yeah but well go for it yeah keep going you're the scientist here we'd love to hear more of your perspective you know oh i was i was very rapidly reaching the end of what i wanted to say. I think probably you all have more to say about the relationship between input and output sizes. Yeah, so go for it. Yeah, I mean, if we're sort of thinking of this predicting the next token thing and this sort of idea of predicting the next word, everybody's like, oh, these GPT models, they're just next word predictors. oh, these GPT models, they're just next word predictors. I mean, Wiz, can you cast this into sort of an input and output context for us, let's say, as we think about the next token and sort of like, how many next tokens can we print out before we start to run into this issue of, I've now filled up my context window. I need a longer context window. How should we think about this? Yeah, I mean, the idea is it's like a tank that can fill up, right? If you add more tokens to the context window by, say, generating them, and then you look at all the tokens that you've had as input, as your system prompt, and that you've generated, eventually you'll have generated enough tokens that you're no longer, you've used all the context window. And something's got to give at that point, right? The models are not built to be hyper flexible in this sense. So we're going to have to tackle the fact that we're going to just start dropping context from the, say, the beginning of our input, right? So we can slide it along. We can continue for as long as we'd like, but if you have, say, a context window of 100, right? And you generate, you give it an input of 50 tokens, right? And it generates 100 tokens. When it's generating that 100th token, it's only looking at half of what you gave it, right? So this is the idea that's kind of floating out of context. And I think that, now I want to be clear, because there are literal hard-coded API limits for a lot of these services that you're going to use whenever you're serving models, whenever you're interacting with API models. Like the API or the application layer is going to put limits on how much stuff you can send it and then how much stuff it can generate. So you're going to see things like maximum input and maximum output, but those are, are, are artificial limits or it, I would say arbitrary, but that feels a little bit crude, but the idea is they are, they're well-reasoned limits that are, that are imposed by people. It's nothing to do with the actual nature of the model. It's just to do with the fact that like, typically, we don't want models to generate for very long time, especially if your business is paying for the compute, right? You don't want the model to generate 700,000 tokens because you're paying for it, and it's probably going to impact latency of other users. And then you're also, you know, you can't let people put in too much because you know, you're going to generate X number of tokens, right. Or up to X number of tokens. So you can't let people put in literally the entire context window, or you're immediately gonna lose context. I'm assuming that you're handling it gracefully even which some, uh, serving platforms don't do. So, okay. So quick follow-up on that. So if I'm generating tokens and then I'm sort of running out of context window, does that mean I'm losing the system prompt? Or can I be more flexible about the way I'm doing this? Is this kind of the way that most of these systems work? I think most of them just don't handle it, to be honest with you. So if you look at the popular serving platforms, they just don't let you do that when the context window gets too full, quote-unquote gets too full uh quote unquote gets too full they'll they'll simply stop working and in fact most of them are designed so that can't happen uh under reasonable circumstances so they limit the amount of input tokens and they limit the amount of potential output tokens so we never reach that point uh but of course with with the brilliant engineering teams that we have uh you could certainly design the system so it preferentially drops the input tokens versus a system. But at that point, what are we doing? Sure, sure, sure. Yeah, yeah. Okay, so then Leo, these long context windows can actually slide and have more input or more output. Is that right? Yeah. I mean, I actually think it might be a little hard to say exactly what would happen if you kind of like give it more context than it was trained on. Like, I don't think it's like necessarily clear what, like, you know, this kind of like goes back to, you know, what a model knows and what it doesn't know. You know, imagine if you yourself have like only ever read, I don't know, 100 page books, and someone gave you like a thousand page book and said, hey, go figure it out. Like you might do something reasonable, but you also might just start doing something completely unhinged, right? And so I think like, you know, trying to put my like product hat or like engineering hat on, I think imposing a clear restriction of, hey, let's keep the context within bounds of what the model knows about. Like maybe is a more like it is a clearer experience or a more expected experience than, you know, go ham and but but God help you with like what the model is going to say. That's right. That's right. Okay, so if the whole point is to be fact checkable and avoid hallucinations, we'd still want to get into this mess. Yeah, yeah, yeah, yeah, yeah. All right. That and just like a point of clarification here, Leo, is it true that like we put all of the tokens that are currently in the context window that is all input and all output each time to predict the next token? Is that sort of what's happening here? Sort of running everything back through each time? Yeah, yeah. There's like caching things that you can get into where like you don't have to like redo all of the computation from scratch. But yeah, exactly. Like you generate the next token by giving it everything that it's seen before. Yeah. OK. And yeah, join us next week. We're talking next token. Exactly how to do these predictions. So I want to go into training now a little bit, and I want to talk about how you guys actually created this long context setup. So like, Leo, I'd love for you to just talk a little bit about the training process and sort of technically, I know you guys had to get access to all this compute. How should we even think about what you did? You took this model that was massive off the shelf, and then you sort of, is this more of an unsupervised pre-training thing, more of a supervised fine-tuning thing, or is this sort of somewhere in between in this sort of continued pre-training space? Yeah, yeah, great question. So in training, kind of like a long context model, there's basically like two main challenges. One is just the pure computational challenge. You may have heard about attention is like quadratic complexity in the context link uh that's exactly because every token needs to be able to reference every other token and so like things that are quadratic don't necessarily scale well and so you like very quickly start like running out of memories these things get huge um and then the other side i think we already touched on is just the fact that like the model has never had to deal with tokens that are, you know, a million length apart. And so like there's probably some amount of like instruction or training. And you need to give it examples for it to like now be able to figure that out. And so those those two pieces, you know, we can get into it in more detail, but, you know, those are kind of like the two things that we had to figure it out. And then the other thing that you asked was about kind of the, you know, the unsupervised pre-training versus the fine tuning. tuning. And yeah, it's a bit of both, right? Like one of the things that we found works pretty well is to do unsupervised pre-training on long contexts, right? Like typically how the pre-training works is you give the model some text, you know, whether it's like code or a book or what have you, and you have it like use the previous part of the book to predict the next token. Do the same thing for long context, right? that's like a very straightforward thing to do um one of the uh issues with with just doing that is it becomes a kind of like a single a single purpose model right like all it knows is how to like give you the next part of whatever you give to it. And so if you want it to then be able to like chat with you, for example, you got to layer on other stuff on top of it. And so so that's what we did. Like we first did the pre-training and then we layered on. It's called like instruction tuning. But you tell it how to chat. That's right. So did you take the base model off the shelf then or did you take the instruction? OK. Oh, no. So sorry. Sorry. Sorry. Take that back. We took the instruction to model. I preempted the question. Yeah, yeah, yeah. Okay. Okay. All right. Yeah, yeah. That's, that's in line with I think all the things that are kind of viewers have heard from us, like, you know, take the instruction tuned off the shelf when, you know, why not when possible? Why not? It knows more stuff. Yeah. It can do more things. Yeah. Can you give us a sense too, of like the, the kind of like problem that you're solving when you really need this long context thing? I mean, you know, tell us Harry's entire life story or something. I can imagine, right. That's a sort of the Harry Potter example, but like, can you give us some sort of sense of what these data sets look like that you're using or the problem spaces that Enterprise is thinking about with these longer context windows? Yeah, they're definitely related. I would say on the on the data set side, one of the things that we ran into is there's just not that many uh like pre-existing long contexts out there like you know at one point we were like scouring like project gutenberg to see what are the longest books out there and it's like yeah like you got like war and peace i think it clocks in at like 700 and something k um I think you have like Les Mis. It's also like 600 K. There's only a few of them out there. So pretty quickly you have to get into like synthetic data generation. And yeah, we can chat about that too. On the use case side, there's a few that that we've thought of um maybe one of the more compelling ones is is on coding um you know for like right now like coding assistants they kind of like know pretty locally like around the the code that you're writing and and maybe like slightly more than that and so they're pretty good at like uh giving you like the the next part of the line that you're writing in Python or whatever. Right. But now imagine all of a sudden you can throw like your entire code base into it. Now we can basically if you get like much closer to this idea of describing the feature you want, maybe giving it enough detail, and it's able to pull in all the stuff that it needs. So that's one use case. There's a couple. Yeah, Wiz, what do you think about that? You think we got the AI programmer coming soon? That is a great use case. I mean, because it's like, I can't really think of things that are that long. You know, it's like the entire code base of something that that's potentially massive yeah so good um anything to add on this Wiz no I think yeah I think it's well covered I mean uh you know the AI programmer is always I feel like six weeks away that's it you know so yeah brought out to Alan and Kevin by the way yeah that's right uh watch out here comes kevin uh we got that on our youtube channel as of yesterday we'll still be six weeks away six weeks from now but i think it is that's right it's better right i mean like uh and when it comes to uh when it comes to something like the the long context uh in this idea of like basically using the the as like a glorified cache. It is an entire code base is a great thing to have it in cache if your company has a very large code base, right? It becomes worthwhile to answer really, you know, really complex questions that require that level of context, you know, quickly. So absolutely. I mean, I can see it being a super, super valid use case. Yeah. Yeah. Okay. Very cool, guys. Well, we're going to go ahead and start the last portion of our presentation today, where we talk about eval next. And what we have prepared a short demo for you guys today because we don't want to make it all panel discussion to give you some some more context on how people are evaluating these llms today a lot of times you'll see for instance this is from gradients post on the 8 billion 1 million context length and you'll see this sort of plot. And this is sort of called the needle in a haystack, where you have here on the y-axis, you have depth percent, sort of how far into the context window are we going? And are we looking for this needle in our haystack? looking for this needle in our haystack. And then we have sort of this token limit, token number that we're sort of looking versus this depth percent. Okay, so we're sort of how far into the different size context windows are we going to look? And you see here, we want this thing to kind of be all green and green of course is this one here. And it means you found it, right? Red down here means, ah, you missed it, right? And so we put this little sort of nugget, this little needle in the haystack, the sort of famous example is the sort of like, you know, kind of magic number. And we sort of just put it in the context and we see if the LLM can find it. So what we did today is we prepared a short demo where we're actually going to see if we can do this needle in a haystack approach on GPT-4 Omni, this 128,000 context length, long context, let's say, LLM that just came out last week. And Wiz is going to present this for us to give you guys a little bit of an insight into how this is happening and then we'll come back and we'll talk about exactly how gradient was doing this with all the models that it came out we'll talk about the future of evaluation and we'll wrap up for more q a in the slido, over to you. Oh yeah, okay, so this is pretty straightforward, thankfully, to some work done by this Greg. The basic idea of needle in a haystack is very straightforward, right? We are going to fill up the context window with stuff, and then we are going to ask a very specific question about something that we've added to the stuff. So first of all, let's just let's just talk about, you know, some dependencies, we need this needle haystack, we need this laying chain community. And we're going to grab the repo that has this, this library, because we also want to yep sorry you guys I will share the the link to the notebook here but the idea is right we're just gonna grab this repo so that we have all of Paul Graham's essays this is like a kind of the the classic example of this right is we use Paul Graham's essays as the haystack so the stuff we're to fill our context window up with is those essays. Now, we're going to need to use this nest async apply so that we can run the async loop in Colab, just necessary. We're going to use OpenAI as both our model that we're testing and our evaluator, right? So in order to prevent situations where the model answers with the correct response, but perhaps using slightly different wording and yada yada, we're going to use an LLM to determine whether or not the response contained the correct answer. So that's why we're using an LLM as the evaluator here. We're going to use a bunch of different parameters. The basic idea is pretty straightforward here. Like we need to make sure that we set everything up and then we can just kind of run this. It is going to take like a while because we are testing a lot. So in this case, we're only going to test a total of 10 intervals and 5 depth percents. So let's talk about what these things are doing, right? So model to test, straightforward. We're going to use OpenAI, and we're going to test GPT-4.0. Our evaluator is going to be GPT-4.0. It doesn't matter that these are the same because they're not doing the same thing. We're just using this evaluator model to see if our response contained the correct answer, so if it contained the needle. The true answer is Flaming Hot Cheetos. The question asked is, what specific food does Jerry like? This is homage to Jerry from Llamaspace, who used this prompt when doing some NIH a little while ago. And then our needle is Jerry likes Flaming Hot Cheetos. So this is basically, the needle is just some text. And the idea of needle and haystack is we fill the context up to some amount with uh you know paul graham's essays and then we place the phrase jerry likes flaming hot cheetos somewhere in that context now the the important piece is that we can place it at various depths which means say for instance zero depth is the start of the sequence and 100% depth is the end of the sequence. And then we can place it at various intervals along that, you know, beginning to end, right? So this is the idea that we want to see, does the model have blind spots potentially in, you know, in the middle, or, you know, towards, towards the end, or towards the beginning, right, this is kind of seeing where does this retrieval task, where is this retrieval task successful. Now, we can also set up the context lengths, so we go from 100 tokens, so that's only 100 tokens of Paul Graham's essays, plus our little phrase, all the way up to 118,000 tokens. 128K is the maximum for GBT4O, but we just chose 118 to give it some wiggle room, as the library sometimes overestimates and overshoots this number. And we're going to split it into 10 intervals. And then we're going to do this, you know, we're going to basically place the token along every 20% as we do the evaluation in that input sequence. Easy peasy. That is it. So the whole idea here is we fill up some amount of context, depending on what interval we're on, with polygram essays, and then we plop that token in at the according depth. And then we see if we can answer this question and get this response. So we run this. It runs for a while. It took about five minutes. And it can cost, right? Because we're just running an API endpoint. Luckily, GBT4O is cheap. So it's not costing too much. And then, of course, we can use our, we can create the table, right? So we'll create a data frame frame convert it to a pivot table and then we're going to use plot lead to plot it you'll notice that there's some blank spaces this is just where uh the the library failed to retrieve a response so they're nans so they show up as clear uh but the idea is and that's the not the model's fault that's the uh library's fault, but this is it, right? So the idea is, and how to read this chart, right, is at 80K, right, at each of these tested depths from zero to 100, the model always answered the question successfully in each of these trials, right? So it didn't matter where we placed the needle in 80K context, it always found it and same with up up to the maximum uh which is going to be 113k is the actual uh value that it used we are able to we are able to uh retrieve or fetch the token in all cases, all the way up to 118K, right? So what this means is that up to its maximum context length, we are always able to successfully answer the question. We'll just scroll up to double check that it's what specific food does Jerry like? And we always get the response, Jerry, or a response close to it, Jerry likes flaming hot Cheetos. So we always have something that says that Jerry likes flaming hot Cheetos. So we always have something that says that Jerry enjoys flaming hot Cheetos. And so what this means is that on this, albeit very simple task, this model is always able to retrieve the correct sequence that we've injected into those polygram essays. Now I do wanna talk about something very quickly, which is that this test is awesome, but it's only testing for that retrieval task. And so we're gonna go back to Greg and we're gonna hear a little bit more about, other ways that we can evaluate and what this truly means for evaluation. So I'll send it back to Greg. Yeah, Flamin' Hot Cheetos are fire. Very cool that Jerry's a fan too. So the needle in a haystack is all about putting something in there and trying to find it, right? We're looking how far in are we able to find it? And then we can sort of expand this idea. The sort of multi-needle in a haystack, we can look at, and this is from a blog that LankChain put out, more needles, right? Longer contexts, you see sort of number of needles is the number of unique facts here, context length, smaller one is green, larger one is red here. And so we sort of see this decreasing performance of how many needles we're able to actually retrieve with the longer context and more needles, right? Makes sense. And we can sort of extend this even further. So there was a paper put out called Ruler, and this was actually from NVIDIA. And the Ruler paper sort of combined the retrieval aspects here, the multi-needle in a haystack sort of retrieval aspects with then this kind of variable tracking approach that was sort of aimed at doing this multi-hop tracing. And so what we're trying to do here is we're trying to look at sort of multi-hop connections. We're trying to look at, you know, a step beyond sort of, Hey, do we found this sort of one thing? And then we can also kind of look collectively and do some aggregation as well, where we're looking at common or really, really frequent words, and we're kind of able to look a little bit more at the long range context through the variable tracking and looking at common and frequent words. And together, these sort of constitute this idea of this ruler approach. And there's a paper link in the slides. It's not popping up right here, but it'll be available post-session. I want to sort of bring our guests back up and kick off the Q&A for today. We're going to stick around for a couple extra minutes to answer some of your questions. But when we think about eval, and Leo, start with you on this, the needle in the haystack thing sort of is towards this kind of perfect retrieval system at least but it does seem that there's this real kind of you know if the motivation is to solve the problem of very long range you know correlations across big context then that's really a little bit different than the perfect retrieval, right, that we talked about earlier. It's, you know, are we evaluating both of these things well today? And if not, what's missing? Yeah, I think the discussion we've been having previously kind of like perfectly sets this up because when you were describing when you're both describing needle and haystack i was thinking about it and like yeah this is exactly a test that rags should just like completely knock out of the park right like literally what rag is designed to do um and so i think you know it's it's a very important primitive for long context, right? Like it's got to get this right. It has any hopes of doing anything more sophisticated, but it is very much the first step, right? Like being able to grab like information in history and replicate a pattern. I think this is like similar to tests folks used to do on language models like a couple of years ago. It's called these like induction head tests. And I think like there was I don't remember the name of the research paper, but it basically showed like a causal link between doing well on needle in a haystack like things and then being able to build up to in context learning so it's like an important first step um but like exactly as you put it for like perfect retrieval and all the stuff that we've been talking about um it's not just pulling out one fact but it's being able to pull out multiple interrelated facts and being able to reason about them um I really like that you talked about ruler because I think it's like 100% a step in the right direction, especially on some of those like later tasks that you mentioned, like the variable tracking, right? You have somewhere in the haystack that like x1 equals 10, somewhere else that x2 equals x1, somewhere else that x3 equals x2. And so like, again, it's got to be able to reason kind of like across these pieces X two. And so like, again, it's gotta be able to reason kind of like across these pieces of information and the same thing with all the other ones too. So yes on ruler and also yes on kind of like bridging that continuum between the very easy things and then kind of like, as you mentioned, perfect retrieval or where we're trying to head with kind of these like real world use cases. Yeah. Yeah. So can we expect like a paper on evaluation from you guys anytime soon or what's going on inside a gradient that is sort of related to this? Are you guys coming up with custom things or are you forced to confront this issue with enterprise? Kind? How's this manifesting for you specifically? Yeah, totally. I mean, I've actually just been talking to the ruler folks at Nvidia very recently. We've been talking about collaborating on a few things. I think they're maybe slightly tangential. But one of the interesting things that that comes up is this distinction between uh like the prompt and and the the task um and and and basically it's that um not every model like expects prompts to be written the same way there's actually some really interesting like uh like Lama3 and how it's actually better to have the instruction at the end of the prompt instead of before the example. And it's actually reverse of what GPT expects. But anyways, if you want to be testing the model's performance on long context, maybe it's like you should be optimizing the prompt a little bit to the model. So we've been talking about that kind of stuff to maybe give like a slightly maybe like clearer or more nuanced picture as to like how do you compare all of these different models. But like for us internally, again, like what we're super excited about is actually like using these models in the real world. And so that's the direction that we've been going. You know, like I think for learning how to improve the models, evals are great. And for comparing models, evals are great. But like I always say, hey, let's let's iterate quickly. Let's let's throw code base into this thing. Let's throw all of Shakespeare's works into this context and see if they can write like more poetry. So that's kind of the direction we've been going in. That's right. That's right. Yeah. Show everything in and then see if the person that does the job now says, yeah, I did a good job on that or not. Right. Very, very cool. Okay. So, you know, just we'll start Q&A now. In conclusion, rag and long context, both necessary. We're talking about this long range, also about retrieval. We heard this sort of four to 5 million is where we hit this diminishing return. So don't expect to see 10 million from them anytime soon, but we're expecting to see great things coming out of gradient and this eval is definitely still emerging. So please upvote your favorite questions. You know, In the spirit of building, I wanted to kick off with one of the questions that I saw from anonymous here that says, my app works-ish. Why should I consider an LLM with a long context? Won't this cost more and take longer? Leo, what do you think about that? Leo Sack I mean, I think it probably depends on the use case. You know, I think if you have an app that works-ish with short context, you know, maybe you don't need to add in kind of like superfluous or unnecessary context and pay for it. I do think there are other cases where adding in a bunch of context is helpful. So it's probably like fairly task dependent at this point. Maybe in the future, once these things get more efficient, once there's more kind of like engineering built on top of it, once the model itself is able to discern maybe a little bit more how much context to use potentially, then maybe the answer is like, just use it for everything. But yeah, I think for now it is a little bit on the user to decide where to employ this. Yeah, yeah, absolutely, absolutely. And I think this gets back to the performance versus cost issue. You know, if you're not gonna increase performance that much, maybe you shouldn't spend more, you know? Yeah, it's like, hey, let's do stuff that makes sense from an application standpoint. So technical question here, Leo, with attention scaling quadratically with context length, for one million context context are dense attention matrices still used or is there some sort of different uh tension setup that you're using yeah um great great question um so the way the way that we trained our models, it does do the full quadratic attention, right? So this is like as granular as you can get, like the model can directly compare every token kind of like on that full context window. And the tradeoff, right, we're talking about kind of like cost versus benefit. And this is, I think, like kind of like the really exciting piece of work that we're like about kind of like cost versus benefit um and this is i think like kind of like the really exciting piece of work that we're like now starting to look into is like you know maybe maybe that's that's not entirely necessary for all tasks uh maybe the context that that's pretty far back uh maybe you can compress that a little bit more right um and and at that point uh you start getting closer to the realm of uh efficiency that that's a little bit near kind of like the short context models um and where this kind of comes up is maybe a little bit less on the training because you know like yeah sure like there are uh you know races to speak, of like getting like the first whatever long context model out. But at the end of the day, you're doing training offline where this really comes into practice is for serving the model. Right. Like if you you probably don't want to give the model a question and have to go take like a 10 minute coffee break every time you do. And so like for these kinds of like actual serving use cases, I think sort of maybe like where this question is implying or like leading to looking at ways to compress information further back in time is like pretty important. Hmm, yeah, Wiz, this is something we've been talking about a lot recently, right? And a lot of people are paying attention to this, you know, with Grok coming on the scene and all this other stuff happening like there's training and then there's inference and like it's different and so like how should we be thinking about this wiz in terms of like long context versus rag training versus inference it seems like we covered a lot of questions today that are in the chat, and maybe this is a nice one to end on. Give us your thoughts, and then Leo, you can sort of close us up. Training versus inference, long context versus RAG. How do we sort of navigate this decision space, thinking about computing and leveraging these tools? Yeah, I mean, so the way I understand what you're saying, I think the big thing to look at is going to be inference, right? So for training compute, unless you're, you know, you're someone like Gradient where your job is serving awesome models to do awesome things, I don't know that you're going to be, like, wanting to spend highly in the training realm. You'll wanna do fine tuning for sure. You'll wanna leverage these PEF methods in order to do that cost efficiently. But otherwise I think you're not gonna have a ton of training budget. When it comes to inference though, you're not going to have a ton of training budget. When it comes to inference, though, inference is a space that is very quickly moving, right? So the folks at VLLM, folks at NVIDIA, of course, folks at even AMD, like everyone right now is, and of course, Grok and these kind of bespoke LLM chips or whatever, you know, they're going to call them. And I'm just going to not mention Google's TPUs, I guess. But the idea is, you know, there are so many different platforms that are doing so much great work right now. I think we're spoiled for choice right now uh and you know it it doesn't take much to get a 10 percent uh you know improvement right in in inference latency so uh versus like this rag uh context window thing i think that's a much more nuanced conversation um where you know we're not seeing kind of those huge jumps in performance. We don't even know how to evaluate base non-long context models well yet, right? That's still not a solved space. We have ways we can kind of do it. And we have golden data sets and this kind of thing that help us to do it. But in reality, we don't have even the tools to evaluate like small, smaller context models well, right? And so this idea that we're then going to be able to evaluate a longer context model super well becomes, you know, like Leo said, step in the right direction with ruler. We're always marching forward, right? We haven't marched back yet, but you don't even know how to evaluate them. And then with RAG, it's the same problem, but now we have six moving parts. So, you know, that's a very nuanced conversation. But the way I think about it is start with RAG and start with whatever your favorite, whatever platform you're currently on has inference optimization solutions, AWS, GCP, Azure, whatever system you're working through, they're going to have a system you can use to run model fast and then start with RAG. If you find yourself running against this wall where retrieval is just not enough and you have a pile of contacts that you need to be referencing I think long context becomes worth the cost very quickly um but it is a anyway I'll stop there yeah yeah and I just want to end like that was sort of a a messy question I think the fine point on it, Leo, is like, you guys released like six models. How should we pick which one to use? And how fast are they doing inference? Exactly. Do we have to wait five minutes unless we get a bunch of GPUs on the million or, you know, most people watching are going to pick up what and then most enterprises are going to pick up what? what and then most enterprises are going to pick up what yeah uh like like really good questions um you know i i think like and this i've been playing around with it recently for for some demos i think on a like you know eight uh gpu l40s cluster um it's doing like the 8B model is doing like 600K context length with like a 10 to 20 seconds, maybe closer to 20 second response time. Take that as you will, right? Like that's something that's like pretty easy to spin up just using like VLLM currently currently so you don't need a ton of like additional coding or magic on top of it um for for kind of like faster and and more efficient inference i i'm gonna say stay tuned like i know um you know we're we're definitely working on that um and in making these models a lot cheaper and easier to run inference on um I know other people are working on that as well. And I think, you know, like, it's a really interesting point of, like, thinking about, like, where to use RAG and long context and pre-training and fine-tuning. The thing that I'll add on top of that is putting examples in context, like in context learning is like way more sample efficient than doing fine tuning. And so like I think as like long context gets a little bit more developed, a little bit more like baked in and efficient, like you have this really cool paradigm paradigm of throw a bunch of examples into the rag, pick out even more of them to more than you could before into the long context and use that instead of fine tuning. And that to me feels like something that's much easier to work with than going through the whole fine tuning paradigm. And so that's personally what I'm like, pretty excited about as far as like upcoming use cases. Retrieve your way into fine tuning through the context window. Something like that. Okay. Very, very like that. Yeah. Yeah. All right, guys. Well, epic discussion. Thanks for sticking around to answer extra questions today. Wiz, Leo, discussion. Thanks for sticking around to answer extra questions today, Wiz, Leo. Really appreciate you guys. Great job. It's time to wrap up today. Thanks for joining us today. If you like this event, let us know. We are interested in getting more awesome speakers and brilliant scientists and engineers like Leo building, shipping, and sharing with us. Of course, like and subscribe. We will be back on YouTube next week talking about the details of next token prediction. It will really connect sort of this big zoomed out view we had today with a very, very zoomed in depth view. If you haven't joined the AIM community yet on Discord, jump in, start building, shipping and sharing with us. We don't bite and neither does anybody else in the community. If you have questions, you run into issues, error codes, go ahead and jump in and ask your question. There's people around to help. And then if you're interested in joining our very next cohort that kicks off next week on Tuesday. This is going to be AI Engineering Cohort 3. We just had an absolute banger of a Cohort 2 demo day last week. Check out those videos on YouTube. That is available. We still have seats left for that cohort. So check it out, get your application started. And even if you don't actually enroll, you'll learn a lot just going through the application process. But that's a wrap for today, everybody. Thanks for sticking around late with us. Please share any feedback that you have about our event in the feedback forms that you'll get post-event. And as always, until next time, keep building, shipping, and sharing, and we, as well as Gradient, will certainly keep doing the same. Thanks, everybody. Have a great rest of your week. We'll see you soon.
Long Context Windows: Extending Llama 3
Join us live to discover how Gradient AI is pushing the boundaries of LLM technology with groundbreaking long-context versions of Llama 3! We'll explore how Gradient's small team outpaced Meta by releasing Llama 3 models with unprecedented context windows, reaching up to 4 million tokens. Learn the technical intricacies of expanding context window sizes from 8B to 70B parameters, the compute requirements, and the challenges faced. We'll delve into popular benchmarks like Greg Kamradt’s Needle in a Haystack and discuss with Gradient's Chief Scientist, Leo Pekelis, the future of RAG versus long-context LLMs. Don't miss this chance to understand the cutting-edge advancements in AI and get your questions answered live. 

Event page: https://lu.ma/longllama

Have a question for a speaker? Drop them here: 
https://app.sli.do/event/9kSLiGTxM1CzkJKmpk3VDS

Speakers: 
Leonid Pekelis, Chief Scientist at Gradient
https://www.linkedin.com/in/leonid-pekelis/

Dr. Greg, Co-Founder & CEO
https://www.linkedin.com/in/gregloughane

The Wiz, Co-Founder & CTO
https://www.linkedin.com/in/csalexiuk/

Apply for our new AI Engineering Bootcamp on Maven today! 
https://bit.ly/aie1

For team leaders, check out!
https://aimakerspace.io/gen-ai-upskilling-for-teams/

Join our community to start building, shipping, and sharing with us today!
https://discord.gg/RzhvYvAwzA

How'd we do? Share your feedback and suggestions for future events.
https://forms.gle/dRMWrwHM9kjGc4A97