 Hey Wiz, we've talked quite a lot about the black art of chunking in our courses over the past six months or so, haven't we? Yeah, we sure have. Yeah. Yeah. People are always asking questions and they're always saying, hey, so how should I chunk my documents? What's the answer? Is there an answer? There's not really a one size fits all answer to the question, even 2024, Greg. Well, there's this new method I heard about and everybody's been talking about recently that apparently asks the question, but what's the meaning of each chunk? It's called semantic chunking. Have you heard about this? I have. Yeah. Yeah. It looks pretty promising, doesn't it? It looks like it looks pretty good. Yeah. Yeah. Yeah. Yeah. Okay. Well, today we're going to dive in and we're going to see exactly how it stacks up qualitatively and quantitatively versus the gold standard. Sound good? Sounds like a great plan. All right, let's do it, man. Let's jump in today. Welcome everybody. My name's Dr. Greg. That's The Wiz, AKA the LLM Wizard. We are co-founders of AI Makerspace. Thanks for taking the time to join us today. Today we're talking semantic chunking for retrieval augmented generation. It's one of the newest methods. A lot of people are talking about it. A lot of the new frameworks are implementing the method. And we're asking the question today, is it something you should put in your toolbox as you start building, shipping and sharing production LLM applications? Today, we're going to get to the bottom of it, and you'll learn if it is something you should definitely keep in your pocket to pull out as you try to take your RAG systems to the next level. systems to the next level. We'll have Wiz back a couple of times today to dig into some details about specific chunking we're doing on the document we chose today. And of course, we'll be back at the end for a demo. If you have questions along the way, drop them in the Slido link for us. That way they can be upvoted and we can make sure that we get to your question. All right, let's get into it, everybody. Today we're talking semantic chunking. And overall today, what we want to do as we align ourselves towards the session is we want to understand, first of all, the process of semantic chunking. We want to contextualize why we would care about semantic chunking for RAG. And then we want to take a really close look at how the kind of best practice chunking method is different, but also kind of the same in spirit as the semantic chunking method. It's all about meaning after all that we're trying to extract when we deal with words and when we deal with generating answers to our questions based on what's actually contained in our private documents. And then we're going to leverage a tool that you've maybe seen us use before called RAG assessment to actually look at how performance differs quantitatively between the two methods for building a simple RAG system. So first off, we're gonna sort of contextualize this thing with a little bit about chunking. We're going to talk about two methods for text splitting, aka chunking. One is going to be the recursive character text splitter, the one that you probably are grabbing off the shelf if you're building LLM applications today. And then we'll talk about semantic chunking. We're going to assess the method and do a complete walkthrough with the RAG assessment. And finally, we'll answer any questions you guys have as we get to the Q&A. So when we talk about chunking, it's really a simple concept. We're going to break our text into smaller pieces. We can't just feed all of the text at once in despite kind of the proliferation of people asking us about long context windows today. RAG is always going to be something where we're chunking. We're always going to be We're chunking. We're always going to be splitting things up into the right amount. And what is the right amount for any given context is a very hard question in general. But it's important to ask ourselves this for our application because all of this fits into the process of retrieval. to the process of retrieval. We need to go find the relevant information that we have in our documents based on any given question or query we might put to an application we build with an LLM. And retrieval, of course, is the first part of RAG. It puts the R in RAG. It's where we find our reference material. It's where we find our reference material. It's where we get our facts that we're going to leverage to improve our generations after we augment our prompt. And of course, putting the R in RAG is just this idea of retrieving often dense vectors. retrieving often dense vectors. And if we take RAG as sort of two component pieces, we can look at the fact that it's really just this dense vector retrieval piece doing a lot of the heavy lifting. And then we're leveraging the concept of in-context learning by putting this information into the prompt, by augmenting the prompt. So it's important to really focus on this retrieval step. When we ask a question, that question is then going through getting chunked and getting turned into a series of numbers. Those numbers are then, we're looking for that representation in embedding space of our question. And we're looking for stuff that's similar to it. We're looking for stuff that in embedding space is close to our question. And that's where we go to our vector store, which is full of embeddings that represent our data that are also in the same embedding space. And we're looking for distance between our question and the stuff we want to retrieve. differently, whether we're chunking our question or whether we're chunking all of the data in our vector store, it's going to change the results that we get. And so this is why we sort of refer to this as the black art of chunking. It's the thing people often don't talk about because there is no sort of right answer, although there are best practice methods. This allows us to then, after we set up our prompt template, return those chunks in natural language after they've been compared in embedding space as reference material. So this dense vector retrieval is the point. Really, it's the thing that drives the improved generations when you finally feed the prompt into the chat model and get your answer. The second piece is the in-context learning piece. It's really not the point of today's lesson. But overall, this retrieval augmented generation starts with retrieval, and retrieval starts with chunking. In other words, what gets chunked gets retrieved. This is the reality of the situation. And so when we look at retrieval, we need to take a really close look at this vector store. This is where we take our data, we put it in in vector format to a place that we can search for stuff. And the thing about the vector store is that the only thing in there is chunks. the only thing in there is chunks. That's it. So chunking really is fundamental here. And so we want to make sure that when we're chunking, we're doing it in a way that we believe is useful. We believe that's going to produce the best possible results. And we believe aligns with what we're aiming to ultimately use our LLM application for. Improving retrieval can be done through different retriever methods, but it can also be done by improving your chunking. Now, Lewis Carroll famously said in Alice in Wonderland, the best way to explain is to do it. We're going to leverage Lewis Carroll's writing today, and we're going to chunk up Alice in Wonderland and put it into a vector store. We're going to do this a couple different ways. Two different ways. But to make sure that we're clear on why those two different ways matter, let's talk about the chunking methods that everybody's mentioning right now. Number one is the fixed size method. Give me your chunk size and number of characters, give me the overlap between chunks. That's it. Okay. Very basic, very simple. The recursive method is the one we're going to dive into a little bit more detail in just a minute. But if this meme is any indication, what we're doing is we're combining some ideas of fixed size. Now it's worth noting that you can use a chunk overlap and chunk size fixed size approach. You could also do something simple like say why don't you just chunk by sentence or chunk by paragraph. There's sort of a natural indication that that's going to be useful in some ways to us as people who are reading and writing in sentence and paragraph form. There's also document-specific type of chunking. This we'll leave aside for the day, but the great use case here is to think about programming code. Oftentimes, there are natural breaks within programming code that indicate, well, this is kind of a chunk of code right here. And so you can do this with different document types. And we won't get into this. As we get into more and more structured type of language data, there's going to be a lot of different ways to potentially do chunking in the future for more complex documents. Today we're focused on sort of unstructured text, more like a novel that we'll use. And then of course there's semantic chunking and we're gonna dive into this in a minute. This is the purpose of today's event. It's also worth noting that agentic chunking is a thing that's starting to be discussed. Now, the idea here is that after initial chunks are created, then you use an LLM to reason about whether or not you should chunk differently. This is sort of the meta pattern here. The idea of agentic chunking, we want to make sure we're clear on is a pattern. And if you missed our event last week on agents for RAG systems, give that one a watch to get a little more insight into the many, many different ways you might use an agentic chunking approach. This would basically be putting some reasoning into the retrieval step. And there is a specific sort of propositional approach people are talking about today with agentic chunking, but we expect that this will be more and more sophisticated. The methods will grow as the popularity with agents and multi-agent systems grows. But for today, we're going to focus on recursive and semantic chunking. In order to understand semantic and recursive chunking, we have to start with this most basic fixed size chunk. Number of characters, overlap, by sentence, by paragraph, do these things make sense? Well, how about if we kind of combine these ideas into more of an ensemble approach. If you remember classic machine learning, this always worked well. And indeed, it works pretty well when we talk about chunking. Introducing the Swiss army knife of chunking. Shout out to Greg Kamrat for coming up with that analogy. This is really the de facto standard. And what recursive text splitting does is it kind of leverages the fixed size, what's good about the fixed size and overlap chunking, an overlap chunking while also leveraging the natural flow of language in sentence or paragraph form. So this is sort of the recommended generic text splitter. It essentially has a fixed chunk size and a fixed overlap in mind. But then it goes through and it looks at different separators. For instance, the double new line separator is indicative of a new paragraph. A new line separator is often is often indicative of potentially a break in the meaning of what's going on. This could be a new sentence, but there's a lot of different ways that authors can use new lines in their writing. If you think about the many different types of authors you've read and how they can get pretty creative. And then as we sort of move through each of these types of separators, we sort of return and we ask ourself about how close we are to our desired chunk size. And we go down through to the space level and to the character level. And I want to look at an example right here from Alice in Wonderland. This is from the very beginning of the book, page two to three. If we use a recursive character text splitter out of the box, this is one chunk that we might get. And I want to bring the Wiz back up on stage now to discuss this a little bit with us. Because so Wiz, we've got this double new line, new line, spacing and character sort of level of separation we're looking at. But yet there's this recursion, there's this return to this fixed size and overlap. Can you talk a little bit about how you understand this chunk being quote unquote decided upon by this algorithm here? Why would it choose this particular chunk? Yeah, you bet. So the basic idea here is that we lack the double new line in this chunk, right? So you can see that there's a double new line before there, which is above this specific chunk. And, you know, we don't have one in the highlighted text. So we can't split on that, right? We're kind of already out. We're then going to see, okay, well, can we split on a new line character? Yeah, well, we have a couple. But we don't have any that are well sized, let's say, right? We have a very small one, we have room left over. So you'll see that we've split here on our space character because of that. Right. So we split at the end on a double new line since we don't want to go into the rest of that paragraph based on the chunk size that we have so far. Right. And then we are we don't have a new line that gets us close enough. So we're going to split on a, on a space character that you can see, like, because these are generated in order. So from start to finish, we left off the previous chunk at O. So we had to start at deer basically. And then the next closest to our desired size, which is 200, was the double new line that's at the end of this chunk. So after again, period, there's going to be two new line characters. That's basically it. So like this, when you start to document, obviously, you're just going to start at the start. And then you go until you find a double new line or your chunk size. And if you find your chunk size, but you don't find a double new line, you're just gonna start at the start. And then you go until you find a double new line or your chunk size. And if you find your chunk size, but you don't find a double new line, you're gonna look for a new line that's close to your chunk size. And if you can't find a new line that's close to your chunk size, you're gonna look for a space that's close to your chunk size. And we just keep repeating that recursively even, until we get our desired chunk. And so this kind of in this specific instance, right? So we were talking about how we're not going to talk about document specific chunking, but because our text is a book and because it has paragraphs and chapters and sentences. It lines up such that the default recursive text splitter is actually a great document text splitter in this case, because it is following the format that we'd expect. Paragraphs contain single ideas, sentences, smaller ideas, and we want to not break on middle of sentence where possible, that's the idea. I see, I see. So really it's not that different than the fixed size. It's sort of shoring up the edges of the fixed size in a way. It's kind of not doing something fundamentally different. It's still sort of aiming at this fixed size in the end. Yeah, absolutely right. The only small advantage is that we're more likely to break or split on a place where you would expect there to be different information on the double new line, right? If you think about a Wikipedia article, right? The double new line is going to be a new section. A single new line is gonna be a new section. A single new line is gonna be interparagraph. And then a space is intersentence and a character is interword. And we wanna avoid interword where possible because those words are definitely a piece of information. And so we'd really rather not split on the word. We'd really rather split on these bigger pieces of information where we're likely to see a difference in meaning between the two. So that's exactly right. Okay. Yeah. Yeah. Definitely not trying to split words up. That makes a lot of sense. Okay. So, but yet this still leaves much to be desired potentially. But yet this still leaves much to be desired potentially. I mean, intuitively I'm reading this and I'm looking at, there was nothing so very remarkable in that. And I want to include this in the chunk. I mean, something about the nature of the paragraph, like this feels like it's appropriate to the meaning of the chunk, doesn't it? It sure does. I mean, in fact, we split the direct quote. And in this case, right, it's maybe not so important because we're just lopping off, oh dear. But imagine we lopped off a negation at the beginning of a quote, right? Like this is something that's potentially problematic. And with the fact that this naive chunking strategy really has no real dynamic size, right? It's just kind of like, it can get as big as it can get. And it's going to greedily try to get as big as it can get. I think this is, we wind up in a situation where we're definitely you know, we're, we're definitely losing something from this quote, like, and again, in this specific example, it's not super impactful, but it's very easy to imagine a situation where that first word was a very important piece of the of the quote, right? So I think it's, yeah, you're exactly right to say, we would love to have that previous stuff, because we know it's related to this later stuff, right? And so we, you know, we can think to do this in different naive ways, maybe we only ever chunk on new lines, and we never do intersentence. So we let our chunk sizes be quite big, or we don't define a chunk size, right? We just know our documentation. But there are many instances where we can't just know our documentation, but there are many instances where we can't perfectly know our documentation or that's going to take too long or too much processing or et cetera, et cetera. So. That's right. That's right. Yeah. I mean, let's say I asked, you know, Hey, who is late? And I returned this chunk. You could probably, maybe a middle schooler could kind of maybe eek this out, but it'd be tough without saying to hear the rabbit say to itself, you know, it's like, boom, there we are. So, okay, very cool. Then I really like that we're sort of seeing there's the super naive fixed size, and then there's the slightly less naive recursive character text splitter. So that's the big takeaway from here. All right. Thanks, Wiz. We'll have you back in just a little bit. So as we take it to the next level here, let's talk about semantically splitting text. Is there a better way? Can we do it in a way that's more meaningful, let's say? way that's more meaningful, let's say. Well, consider a classic retrieval algorithm improvement choice that people building with Langchain will often make today. This is something that we've used extensively in our courses and that we encourage folks to check out as a entry-level advanced retrieval method. It's called the parent document retriever. And there's two big ideas in the parent document retriever. One is that small documents are good, and the other one is that big documents are also good. Let's focus in on why small documents are good right now. They're good because the embeddings accurately reflect meaning and relevancy. Small is good because the embeddings accurately reflect meaning. This is key because if we can accurately reflect meaning in our embeddings without having to make them small, then we solve the same problem that small docs within the parent document retriever are solving. This is exactly what Semantic Rag is trying to do. In other words, as we look at big documents are good because they retain context within each chunk and small documents are good because embeddings accurately reflect meaning semantic rag is going to accurately reflect meaning and retain context within each chunk at the same time. Semantic rag, one way to think of it is as the best case of a parent document retriever. Now we've got a event in a couple of weeks on advanced retrieval. If you want to see us break down a ton more of the state-of-the-art methods, definitely check it out and join us for that. As we look at chunking semantically, and shout out again to Greg Comrott, he's kind of the guy who came up with this idea, and he came up with this idea based on a tweet from Linus here. Greg actually went and did it. Very cool. And he said the hypothesis is we can use embeddings of individual sentences to make more meaningful chunks. So we're going to chunk by sentence. We're going to look at embeddings of each sentence chunk. When we create chunks, what we'll do is we will, number one, split the document into sentences. What we're going to do is we're going to group sentences as well. So we're going to take sentence one, two, and three. We're going to compare it to sentence four, five, and six. This is the standard in the method. You could imagine doing this with groups of any number of sentences, of course. We ask ourselves, is the block of sentences one through three close in embedding space to the block of sentences four five and six how close how similar is that similarity metric if they are similar right if they are similar we want to combine those into one chunk because if they are similar, we want to combine those into one chunk. Because if they're similar in meaning, we want to use why small docs are good and combine them. They're good because they capture meaning well. So if they're similar, we want to combine them. If they're too different, right? If they seem to mean things that are too different from one another, then we'll split. This is the big idea. Now, if you've got a good intuition on this, you might think, well well we might get some pretty freaking large chunks if we do this and in fact you'd be right because out of the box when we implement semantic rag from langchain we get a massive chunk again we're not chunking here based on fixed size and overlap. This is the beginning of the book, and this is on page six, well, at least of the edition we grabbed offline. it goes all the way from the beginning to page six, using this embedding similarity chunking approach. So I want to invite Wiz back up here again to talk a little bit about how he's thinking about how this thing captured six pages in one chunk. Is that right? Is that what we got here, Was? How did that happen? I mean, yeah, it kind of makes sense, right? Like, the chapters are ultimately a large semantic chunk of information. That, you know, likely, what we're gonna think of as a single semantic related portion of a book right if you if you asked someone you know hey how would you best group chunks of meaning in a book they're probably gonna say oh chapters is a good start right usually it's a coherent idea now there's there could be some difference. And in this case, there is some difference, right? So we see like a distinct shift between the two chapter or the two sections of this chapter, one, which is before Alice starts to fall down the rabbit hole, and the second, after which she starts to fall down the rabbit hole. And, you know know i think this is why we have this break you know to try to guess at the uh the reason but the idea is that the first part of the chapter is basically just about alice wandering around bumping into the rabbit right and uh all of that information is relevant to to this chapter and so that's why we get this, this split here. And that's why we're capturing so much information, right? The idea, especially when we talk about long context window and everything above, blah, blah, blah, blah, blah. When we talk about these things, we are we are left in a place where we're thinking about, well, if this is all related to the first part of this chapter, if all this is interconnected, right, if this whole first bit is about Alice to the first part of this chapter, if all this is interconnected, right? If this whole first bit is about Alice meeting the rabbit and then falling down this rabbit hole, then we should group it together, right? We should provide that as context. We're most likely to include relevant pieces of information if we do that. As well, you can think of this as an analog to parent document retriever, as you suggested, except instead of just relying on locality, right? So we know that pieces of information that are near each other are probably related to each other. Instead of just relying on that concept, we're also going to leverage, well, we should think about if they contain meaning to each other, right? It's like in a textbook, if you're talking about one concept, and then you pivot to the next concept, yes, the locality would would would would lead you to believe that these things are related, but they're they're different in meaning, right? The first bit might be about a certain, you know, a certain situation, and the second bit might be about another situation. And so, you know, this lets us build these larger chunks, without compromising the meaning of that of that chunk, or of those individual chunks. The other way to think about it that I like to think about it is whenever we embed something into this space, we, you can think about it that I like to think about it is whenever we embed something into this space, you can think of it as literally a point in 3D space or n-dimensional space, right? And what we're looking at is every time we add a new set of documents, the point's kind of in the same place, right? And that's true until we shift it past a certain threshold, right? Which is what we're going to talk about a little bit in the code. And if we move it past that threshold, we're saying, okay, there's a difference now. But as long as we keep kind of pointing to the same place in that space, I mean, all that information's, you know, wonderfully relevant. And it will only help us to understand the context. And so that's how I think about it. Yeah, I really like this textbook idea or sort of the technical report idea. It seems like in those cases, there'll be much more natural shifts in context that you might not find in a novel. I mean, fundamentally, right? a novel is kind of all the way connected on some level. And so, you know, to sort of do subchapter even is tough. I mean, if in many novels, there's subchapter headings, and that's the natural break point, right? And so, you know, we could, we could, you know, sit here and get very, very particular about, well, if it was up the rabbit hole and down the rabbit hole, as you say, well, why did we get down rabbit hole and down the rabbit hole as you say well why did we get down down down in the first you know and and there's all sorts of really nuanced things when we try to be data centric here but this is a very useful way to to think about especially in specific types of documents i I think. Love the textbook idea, love the sort of technical reporting idea. Places where you sort of switch context more than a novel, this might be incredibly useful. And, you know, that to me feels like a potentially big, big win for this method, although I don't know. And we'll see what the numbers tell us here shortly, won't we? We sure will. Yeah, yeah. All right. Well, let's go ahead and introduce this build. We'll have you back in just a little bit, Wiz. What we're going to do is we're going to now see if we can compare these quantitatively everybody likes numbers it's hard to compare these things qualitatively for any given type of document in general as I hope we've sort of demonstrated with this simple novel what we're going to do is we're going to load the document. We're going to chunk using two different methods, the ones that we've talked about, the recursive character text splitting and the semantic chunking. We're going to, in the notebook, see if we can look at an example or two from Alice in Wonderland. Is it clear which one's better? Well, we kind of talked about the primary example of the session. I think it probably makes a little more sense that we kind of split the chapter in half, maybe up the rabbit hole, down the rabbit hole kind of thing, versus being more rigidly attached to a 200-character chunk approximately, but one that's looking for new lines or spaces between words. And then by building out a simple laying chain rag system, we're going to be able to quantitatively wrap that in the rag assessment framework and decide what the metrics we're getting are telling us. Of course, we're using Lewis Carroll's Alice in Wonderland to do our chunking. We're using OpenAI's models here. And we are using the RAG assessment framework, which we've talked about in a recent event on the Art of RAG evaluation. If you want to go into details on the latest from the guys over at RAGIS, shout out to them and their recent Y Combinator appearance, then check out this event to dig into all the details of the RAGIS framework. But for now, you'll understand kind of what's going on at a high level as we see the numbers go up or down as we assess semantic chunking of rag quantitatively in the notebook Wiz sending it over to you oh yeah okay so we have uh pretty straightforwardly you know the the way we begin most of these builds we're gonna get some dependencies uh we're going to get some dependencies. We're going to grab lane chain experimental, which is new. That's because this is an experimental feature still being worked on. Not everything's ironed out. Some of the strategies are still a little bit naive and this needs to be, you know, built upon or, you know, expanded on in order to reach like the full, quote unquote, the full power, right? So we're going to keep that in mind as we go through the notebook. We're also going to grab OpenAI since we'll be leveraging those models through both our RAG LCL chain as well as our RAG assessment. We're going to just grab Langchain Core and RAGUS. We're also going to grab Face CPU and Tiktoken. Face CPU is just what we're going to use to build our local embeddings store, so our vector store. We're just going to use Face to power that. Next, we're going to grab Alice in Wonderland. We just grab it from Gutenberg. This is a copyright-free book, so it works for the demo today. We're going to read it into memory, and that's it. We're just going to leave it there. You can use a document retriever from Lionchain, but there's not a significant difference here. So we're going to first just chunk these naively to see what the chunks look like, and we're going to see that we get our recursive character text splitter. We're going to have a chunk size of 200. No overlap. We are going to use the Python len function which means we are not using token length here. We are just using characters. And then we are going to keep the is separator equal to false. This is just a parameter that you should set. We are then going to split our documents and look at some of the chunks. We can see, right, this is exactly what Greg outlined to us. We have a number of small chunks, and these chunks are broken apart kind of just, you know, in the middle of sentences. They're broken apart, you know, maybe inconveniently. Again, this chunk size was picked purely to illustrate this point. This is still true at a larger chunk size. It's just going to be less frequent depending on how many documents you have. But the idea is pretty straightforwardly, right? These chunks don't contain a lot of information in each chunk, and it's disjointed from potentially relevant information nearby. So how do we implement semantic chunking? Well, first we're gonna need your OpenAI API key. You can use a local embeddings model for this, like any of the Hugging Face models, you use Cohere's embeddings, whatever embedding model you want, we just do need something to embed our chunks. So we're gonna look at semantic chunker a little bit more technically here. So we kind of got the intuition of how it works and why we care about why it works or how it works. But we're gonna talk a little bit more about the actual implementation. So the implementation is very straightforward. First of all, we're gonna have a number of different thresholds uh these descriptions are straight from the lane chain dots uh you can see here uh I just you know nothing explains it better than the actual documentation so there you go but the idea is you know when we think about when to actually split right right? When to make a gap in chunks, we want to do it because there's some kind of difference. So the first thing that we can think about is a percentile difference, right? So once we get all of our differences in between our sentences, we can see, you know, if there's an X or greater percentile split, that's when we make a break in the chunk, right? We can also use the standard deviation. So if the chunks are greater than a certain standard deviation away from each other, then we should not incorporate those together. And lastly, we can use interquartile distance between the chunks to determine when they should be split. These actually have a significant impact on how your document is going to be chunked. So I would experiment with all of them. In this case, we're just going to leave it as a default, which is percentile. So if we have, you know, these greater than a certain percentile difference between two chunks, we will not combine them together. And the way that this actually works in the code behind the scenes is we first split our entire set of documents into sentences. We're going to do this very naively, just based on period, question mark, exclamation mark, right? So we're not doing like spacey sentence tokenization or anything like this. We certainly could. And maybe that would lead to different results. Maybe that would be better. Certainly be more compute expensive, though. So we're just going to naively, you know, break on sentences as we understand them based on these punctuation characters. We're going to index each sentence based on its position in the document. So the first sentence is the zeroth sentence. And the last sentence is the number of sentences sentence, right? So pretty straightforward stuff. We're just going to index them based on their position. Then we're going to do something that I think is integral to this strategy. We're going to combine sentences without thinking. No embeddings required. We're going to combine groupings of sentences. By default, we're going to use this buffer size equal to one, which means that the sentence that's distance one on either side of our target sentence is going to be included in a group by default. So as Greg expressed, right, sentences one, two, 3 are considered a single unit. Sentence 4, 5, 6 is considered a single unit in this specific example. 0, 1, 2, and 3, 4, 5, if we want to get very comp-sci about it. But this buffer size is something you can play with. The basic idea here is that we expect that the sentence by default will be related to the sentence before and after it. This is not strictly true, doesn't have to be true, but it's a good thing to think about. And it's a good place to start because we do want to make sure that we're capturing some, you know, combined representation of these sentences, right? We don't want just each sentence because each sentence might be very different from each other, but still related to the whole subject matter. So we're going to seed, seed the embedding with this, this kind of three sentence idea. Then we're going to calculate distances between groups of sentences. We just look at their embeddings, and then we calculate the cosine distance between them and that's it. Now we have a bunch of groups of sentences and we have the cosine distance between each of those groups. Then we're going to merge the groups based on the similarity, based on the chosen threshold, right? So this is where we're making the decision. If sentence group A is X percentile different than sentence group B, we will not merge those together, and so on and so forth. And so that's the idea. Again, this is experimental. It's still being worked on. We still have a lot of space to explore here. There's not a ton of research that's concrete on this. So, you know, expect that this will change and become more and more performant over time. The actual implementation is quite straightforward, right? We call a semantic chunker, we provided embedding model, and we choose the breakpoint. If you don't choose the breakpoint, it will default to percentile, but you can choose it. So we show that here. You'll notice that we're gonna use text embedding three large. This is intuition based only, but my assumption is the better our embedding model is, the more accurately we're going to be able to chunk things based on that embedding. So I went with a large embedding model here, though you could choose a smaller one. So you can choose a tech embedding three small. It's totally up to you. I just went with the intuition that a better embedding model will lead to better chunks. And I have no concrete you know there's no research that says it's true it seems kind of likely so that's why we did that uh and then you can see that we get this chunk exactly as we saw from greg uh you know we it's quite long it's it's in fact 4 000 characters right so that's uh you know 4 000 characters is quite a lot each of our other chunks only has 200 characters. And so if we have, if we do the math, right, you know, we're going to see that we have the equivalent of like 20-ish chunks. And that's, you know, that's something that we'll keep in mind going forward. So now we can create a RAG pipeline. We're going to use TextEmb 3 large for our vector stores embedding model. There's no research that indicates, you know, that we should use that. But the, again, the intuition would suggest we should use the same embedding model for comparing our query vector to what we have in our vector store. That we did to create the chunks, right? That's going to, that will communicate the best semantic consistency between the two different processes. So that's why we went with that. But again, you could use anything. I mean, it will work if you use anything, but we chose this with the intuition that, you know, because we created the chunks and the chunks are related to each other semantically according to this embedding model, then we should use the same process to compare them. And then we are going to just use one chunk here, right? Because they're big chunks. And so it makes sense. We'll just use one semantic chunk. And then we can ask questions, like who has a pocket watch? And we get some kind of response. Then we're going to create our, uh, uh, augmentation prompt. Then we create our model and then we create our chain. This is all LCL stuff. Uh, we got a ton of, uh, a ton of, uh, you know, previous material on this. If you're, if you're interested in exactly how this is working, but, uh, for now we just create the chain and then we can ask questions. How does Alice find herself falling down the rabbit hole into Wonderland? And we get this awesome response. It's quite fully fleshed out, pretty cool. And then of course we ask about Dina and we get Dina's Alice's cat and she is important to Alice because she is a great mouse catcher. That's pretty cool. Then we can go to the other strategy, which is naive. We're going to use the same embeddings model just to make it fair. And then we're going to use 15 chunks of retrieved context. The idea here is we want to level the playing field a little bit, right? So where we have these one big chunk, we're going to use 15 smaller chunks uh in order to give this a chance right and the idea is that this should make it a more fair comparison as a token count should be relatively consistent between the two methods uh then we're gonna ask the same question and we get a slightly less robust answer uh and we get a an answer about Dina, but it focuses more just that she's great. And it doesn't focus on why she's great and why she's great because she's a good mouser. So, you know, we lose that information. Okay. So we, so like, I mean, just looking at it, you can kind of say, okay, you know, maybe this is actually preferential. These responses seem a little bit more relevant to the answers. Okay. Let's see if we can actually put that into numbers. So we're going to go ahead and we're going to create our ragas assessment comparison data set. We're going to chunk the source document in a different chunk size, just so our naive rag can't just cheat, right? It can just cheat if we use the same size. So let's not let it cheat um so we're gonna create it in chunks of 400 you can choose any number here uh it's uh just just as long as it's meaningfully different from your uh your other strategies right your other pipelines uh because we don't want it to be able to cheat then we we're gonna create questions, which are synthetically generated by GBD 3.5 Turbo. This is for each chunk in our data, right? We're gonna use those contexts that we created above. We're gonna create ground truths, which will be generated by GBD 4 Turbo. And then we're gonna have our rag chain answer the question based on the question and context, though it will retrieve its own context as that's the point. So we have this here, question prompt, you're a teacher preparing for a test, please create a question and using the following context and question to answer the question, only the providing context. There you go. And we'll create these two chains and we'll run this through. You can also use Ragas' built-in synthetic data generator. You know, it is a wonderful tool. We're just showing kind of what's happening under the hood here. And then we will create our data set. And remember, the question is generated by GBT 3.5 Turbo. The answer comes from our semantic rag pipeline. The context comes from the ground truth context and the ground truth answer comes from GBT4. Then we're going to evaluate based on answer relevancy, faithfulness, context recall and context precision. Just the defaults from Ragas's documentation. And we'll get some results. And I mean, we see the result, context precision is high, faithfulness. We have like a hit or miss. We get some zeros and some nans here. So not a numbers. Our answer relevancy is pretty decent at 0.73. And our context recall is quite high, which makes exact sense. So that's great. Now let's look at the naive strategy, right? So same thing, but the naive strategy, and we can see that they're a little bit different. And I'll let you guys go through the numbers in detail in notebook if you wish. But we can see that the actual naive result is pretty much the same right for everything except for answer relevancy and our answer relevancy is notably worse with the naive strategy which kind of makes sense right like it has less of that relevant information and so it's able to answer the questions less fully and uh and there you go that's the that's the basic idea. And we see the semantic chunking strategy seems to have better performance on answer relevancy without giving anything else up. And so it seems like a good idea to pursue it with perhaps some modification and perhaps more testing on which thresholds work best. But the idea is, if you're to ask me, is semantic chunking better than naive chunking? I would say to you, seems so, yes. That's what the assessment seems to indicate. So with that, I will push you guys back to Greg, who will take us to our Q&A. All right. Thanks, Wiz. So does what it says on the tin. All right. Well, excellent. That was semantic chunking of rag. And in case you didn't quite catch those metrics, just briefly, context precision is asking how relevant is the context to the question. Context recall is asking, is the retriever able to retrieve all of the relevant context? And we got similar performance on both of these. Faithfulness is asking, is the answer fact checkable or is it a hallucination? Again, similar performance here. But the difference was on answer relevancy, which asks how relevant is the answer to the question? Interesting. The semantic chunking process of splitting, then indexing on position before grouping, calculating distances between those groups, and then deciding whether or not to merge based on similarity thresholds. This is something that we'll continue to see evolve. And it does appear that it produces good results in some cases. As we think about this particular key result of the day, the answer relevancy improvement on semantic chunking versus recursive text splitting, this is a generation metric. And again, it asks how relevant is the answer to the question? And again, it asks how relevant is the answer to the question. Importantly, this does not consider factuality, but instead penalizes cases where the answer lacks completeness or contains redundant details. Specifically to calculate this score, an LLM is prompted to generate an appropriate question for the generated answer many times. And then a mean cosine similarity between generated answers and original question is measured. So that overall, if we can generate an answer accurately addressing the initial question, Overall, if we can generate an answer accurately, addressing the initial question, the LLM should be able to generate questions from the answer. Little Uno reverse card metric here. And we see the improvement. So that's pretty cool. More relevant answers with semantic chunking. And that's kind of the takeaway for the day. Remember, what gets chunked gets retrieved. So if you're doing RAG, you should think about chunking. You can be naive, and that's fine, but maybe you can take it to the next level. And maybe you should keep this in your toolbox and watch the emerging techniques, especially the thresholds at which we do semantic chunking. It was quantitatively better and perhaps it was even qualitatively better. Perhaps it could be even a little bit qualitatively better on different types of documents that are suited for this. And we've got a couple questions I just want to ask. Welcome back up to the the stage Wiz um you kind of mentioned this before but I do believe that it's really kind of worth tripling down on does it matter which embedding model we use when we're going and we're thinking about chunking or should we always just kind of you mentioned bigger and better model likely going to be more useful in this case. That's sort of your intuition. Should we be playing much with the embedding model as we're playing with semantic chunking? Yeah, I mean, so I feel like the answer should be yes to this question. We'll have to wait for more kind of results and research for for that to be upgraded to like a yes definitely that's true uh you know i'm not just saying it because it feels right um but it does feel right right so the better the better chunking we have uh in this case is related to how well we can capture semantic information about our about our sentence groups so the better we can represent those sentence groups the better chunking we should have right the smaller differences we should be able to capture and so i think that's uh i believe the embedding model should matter. Similar to how chunk size matters in, you know, naive chunking, right? Or the characters you split on matters. I think it's the same for the embedding model and its performance here. Because we're actually using it to do the chunking. That's correct. That makes sense. So it's literally directly dependent on it. Okay. So I want to go through a couple of questions here briefly. One super common, here we go. Are there better alternatives than cosine distance? What's up with this cosine similarity? Can we get better than this? Man, that's a tough one. I mean, the answer is yes. But there are everything, everything that might be better is much more compute intensive. And in that's the kind of the game we have to play with these distance measures is we have to do a lot of them, right? So we have 500, you know, sentence groups. We have to compare each of those, right? That's a lot of calculation that needs to happen. And so for it to be very efficient and performant is very important. And I don't know that we have something that's drastically better than cosine similarity that would work well in that slot without blowing up compute costs. That being said, it does kind of depend on your data and you could use some of the other simple distance measures if they work better for your data. But realistically, we build everything with this idea of cosine similarity into it. And so we want to keep using it. Yeah. And if you want to do a sensitivity analysis on all the different metrics have at it, I think there's a lot of people working on their PhDs doing this kind of thing right now. All right. How compute intensive, speaking of compute intensive stuff, is this strategy in general? Would it be appropriate in the scenario where we have to constantly update and re-index stuff or not really? Okay. So that's an interesting question, right? If you have changes in documents that you're re-indexing, then this is going to definitely be not the option for you. If you're adding new documents, I mean, it doesn't matter, right? We're only doing this in each document, right? Whenever we add new documents, we have to do this process once, but then it's done. And so if you're working with dynamic documents that are themselves dynamic, so they change, this might not be a great strategy in terms of cost. If you're talking about documents that are added, so you have dynamic documents, but the dynamic part is that you add or remove documents from the pool, that's a great strategy. I mean, it is always going to be more expensive than a naive strategy because we're having to use some embedding model, even if we run it locally, even if we run it on CPU, it's going to take more clocks to get it done than you would have if you were just checking based on which character was being split on. So yeah. Okay. All right. Cool. cool. So, you know, we talked similarity, distance measure. We talked embedding model. The last thing, we get this question all the time. We'll end on this. What chunk overlap should I use, Liz? Like, what's the answer? What chunk overlap is the answer? You should use the one that's best for your documents. You should use the one that's best for your documents. And you should determine that through tools like RAGIS or any other evaluation framework. You know, chunk overlap is a hyperparameter. We should do hyperparameter tuning of some kind. You know, in this case, maybe not literally hyperparameter tuning, but an analog. We should empirically determine the chunk overlap as we should the chunk size by running some kind of search and choosing the one that makes the numbers the highest. And there you have it, everybody. Wiz, I believe we have one more ask of everybody today, don't we? Yes, that's right. I forgot to say it. Please, it helps us a lot if you subscribe to us on YouTube, as well as click the little bell notification to get notified of our future events. We do one every Wednesday at this time. You can always find us here. So join us for your lunch break if you're EST. Otherwise, you know, thanks so much for tuning in today. Thanks, Wiz. Awesome, man. And it's time to close it out, everybody. If you like this session, definitely ring that bell. But also consider joining our Discord. We've got great vibes and we'd love to have you. It's growing rapidly and there's always some interesting conversations going on every day at this point. If you're looking to learn more for free and get started with production LLM applications, check out our open source course, the one we taught last August on LLM Ops cohort one. You can find that on GitHub and we've got a number of YouTube videos associated with it. Of course, if you're ready to accelerate all the way out at the edge, we are happy to announce that we've got our upcoming cohort two of our AI engineering bootcamp that kicks off next Tuesday, April 2nd, and it's going to be the best yet. We've been rapidly iterating in cohort one, which wraps up this week. Check out LinkedIn for the demo day event that you could join us live for on Thursday and check out what people have been building, shipping, and sharing. And finally, if you have any feedback for us, we'd love to have it. Let us know what you'd like to see next. Let us know how you thought these events have gone recently and what you're paying attention to out there from chunking to retrieval methods to agents to multi-agents and beyond. And as always, everybody, until next time, keep building, shipping, and sharing. We'll continue to do exactly the same. See you back next Wednesday or maybe tomorrow night for AIE1 Demo Day. Cheers, all. Have a great week.
Semantic Chunking for RAG
In this event, we’ll learn how the semantic chunking algorithm works! Text is split into sentences that are converted vectors through an embedding model. Similarity is measured between each pair of consecutive sentences. If sentences are too similar, as defined by a threshold, additional chunks are created. We can ensure that if any two consecutive sentences are too different from one another, additional chunks can be created. In theory, this will allow us to achieve better results during retrieval within our RAG system.


Event page: https://lu.ma/chunkingrag

Have a question for a speaker? Drop them here: 
https://app.sli.do/event/eQnuJrAp9sN3RrhMUvMfMS

Speakers: 
Dr. Greg, Co-Founder & CEO
https://www.linkedin.com/in/gregloughane

The Wiz, Co-Founder & CTO
https://www.linkedin.com/in/csalexiuk/

Join our community to start building, shipping, and sharing with us today!
https://discord.gg/RzhvYvAwzA

Apply for our new AI Engineering Bootcamp on Maven today! 
https://bit.ly/aie1

How'd we do? Share your feedback and suggestions for future events.
https://forms.gle/1UxkU7LbfV14f77p7