 Hey Chris, did we find anything curious about doing stuff with complex PDFs? I would say so, yes. We did find a few curious things. It turns out, unsurprisingly, that it's difficult to do things with complex PDFs. And the tools that Lamedex has today are pretty good for a lot of different file types, but still fall a bit short for PDFs. Yes, I believe industry has run into this issue. The question is, does the embedding table unstructured retriever pack do what it says on the tin? For the most part, yes, especially as long as you stick to the kind of suggested file types. It does a great job. Absolutely. Oh, man, definitely looking forward to that. So the exclusive, silky, smooth, perfect one-click auto magical solution for PDFs, it still doesn't exist. And we knew that coming into this, PDFs can get crazy complex. It seems like this is still a work in progress for the industry. Is that right into this, PDFs can get crazy complex. It seems like this is still a work in progress for the industry. Is that right, Chris? That's ultimately correct. Yes. Yeah. Okay. Well, I'm pumped to see how far we can go with what data types and how we can build pipelines with our own PDFs in a more streamlined way than ever before. That's what you've got in store for us today, right? You know it. Well, I'm really looking forward to this one. I'm going to outline for everyone all the core constructs we need to start building with these tools and with these ideas. We'll see you back in just a little bit to lead the data demo. All right, everybody. Thank you for joining us. Myself, founder and CEO of AI Makerspace, and Chris Alexiuk, the LLM Wizard, co-founder and CTO at AI Makerspace for another YouTube Live event. If this is your first time, welcome. We'd love to hear where you're joining in in the YouTube Live chat. What you'll learn today is you'll learn how to build RAG systems with really pretty awesome indexes for your data that contains both tables and text. Special thank you to Llama Index and unstructured.io for partnering with us on today's event. We're really excited to highlight their tools for all of you today. If you have a question throughout today's presentation, please drop it in the Slido link in the chat description or go ahead and click the link in the YouTube live chat right now. All right, let's get into it. So let's talk RAG for complex PDFs. What we like to do at AI Makerspace is we always like to align our AIM for the session. If you stick with us, what you'll get out of today is you'll understand how the core constructs of Lama Index stack up to advanced retrieval methods and how we can sort of build these ideas like Lego blocks in our mind so that we can deal with more and more complex LLM applications. You'll learn how to deal with unstructured text, natural language data, and structured tabular or table data at exactly the same time using this very, very cool unstructured.io functionality that Lama Index has wrapped and made easy for all of us to use. So we're going to talk first about RAG as an overview. I know we've got a lot of new folks in the room today, so we're going to do a quick overview of RAG. We're going to look at some Lama Index core constructs, but we're going to get into the PDF problem as quickly as we can today. We've got a lot of great stuff to show you. So RAG, it's just important to sort of put this into perspective as we get started. It's trying to solve the problem of hallucinating that LLMs have. They're confidently telling us things that are false. This makes people very uncomfortable, especially in enterprise. And the way to solve this is the same way we solve it in real life we want to make what we're doing fact checkable we want to make sure that we're providing references for everything that we say we want our llms to do the same thing references matter and so retrieval augmented generation is all about those references that matter in improving our generations by using them. Retrieval is about finding references. Augmentation is about adding those references to your prompts. And then generation is about getting better answers because we added those references to your prompts. And then generation is about getting better answers because we added those references to the prompts. That's all there is to it. Retrieval, augmented generation. Now, this is useful as a first cut. Very, very useful in specialized domains where you have lots of jargon and lots of words that maybe have slightly different meanings than they do in other contexts and our example today is going to be in the financial domain domain so you'll notice that this is another good candidate for things that you want to try to start teaching your applications ways to understand that specialized language. Of course, you can fall down the rabbit hole of specialized domain language and start fine tuning your embedding models. We did that in a previous Llama Index event. We won't be doing that today, but we will be hopefully able to engage with some specialized language that's in the PDFs we take a look at. RAG from a system architecture perspective can be looked at a few different ways. I like both the linear visualization and the sort of high level viz from Lama Index. The idea of you have to build your indexes first. You're going to load all your data. You're going to figure out how to build the indexes, you're going to get everything ready to rock and roll before you do quer these generations, you want to be able to evaluate are these high quality generations, how can we improve them, are there things we should be measuring, etc. So I like this also just because it sort of shows a lot of different data types types that all go into this idea of index or indices or complex indexes that are managed in very, very unique ways. This is really where Llama Index shines as far as we can tell brighter than anybody else in the industry is when you need to do complex things and really, really get your data into the right order for the best possible results. So we ask a question, we look amongst our index or indices, we add that relevant information to our prompt before we go ahead and put it into the LLM and get hopefully a better answer. We go ahead and put it into the LLM and get hopefully a better answer. One of the ways we like to visualize this is we ask a question. We provide a query. That query is fed to an embedding model. That embedding model then turns our query into a vector format. That vector format is then fed into a vector database. And we're looking for similar things. This is just a vector similarity that we're looking at, something as simple as cosine similarity to get nearest neighbors. From there, and it's important to note here that even when we have quantitative and qualitative information like we'll have today, ultimately everything is being converted to vector format in the examples we'll see today. And so this overarching idea of looking for similarity within a vector database is still holding. You can imagine as you get much more complex in the way that you're storing different data types, perhaps it's not always vector format, but it is for today's examples. And then we're going to go ahead and set up our prompt template. Our prompt template, we might make sure says something like use the following context to answer the user's query. You may not answer the user's query unless there is specific context in the following text. If you do not know the answer or cannot answer, please respond with I don't know. Some good old fashioned prompt engineering there to tee us up for when we do find those similar reference materials, we shove them right in to the prompt and hopefully improve our generations. So the simple retrieval process, the simplest retrieval process, a big idea is you ask a question, you search a database for stuff that's similar, you return that similar stuff. And of course, since we're dealing with vector databases, we're going to talk about taking our documents, splitting them into chunks, creating embeddings for each chunk, and then storing those embeddings in our vector store. So now that our documents are sort of PDF level documents, let's say, are being chunked, we're just sort of doing something a little bit more complex here, where we're taking the data, that's language data, we're taking the data, that's tabular data, and we're chunking and indexing in very powerful ways. And we're chunking and indexing in very powerful ways. And we're providing very powerful patterns that we can leverage in our applications that we are building today with Lama Index. Lama Index, again, it's really shining on the data side as far as we can tell. And that's represented straight top level in their documentation, a data framework for LLM applications. It's all about the data, the data-centric AI paradigm does not go anywhere just because of LLMs. So LLM Index is all about understanding that LLM applications shine the most brightly with private and domain specific data. As you flavor them, that's how you get your competitive advantage. Of course, data is often all over the place in organizations. The bigger the org, the more all over the place the data is, generally speaking. And getting it all to the right place for the right tasks and for the right applications, that's what Lama Index helps us do. So to understand this new capability, we have to understand the core constructs of Lama Index. The first of which, perhaps the most important for us to understand and get familiar with the language of are nodes. Now, nodes are, quote, first class citizens in Lama Index. They're just chunks of source documents. Okay, so we talked about chunks. A node is a chunk, but node is a specific Lama index term. Node is going to inherit the metadata from the parent document. In this case, document means sort of PDF level document. And in this case, we're doing just a simple query where we're looking for the top K nodes that once we find those similar nodes to the query, they'll be fed into a response synthesis module and we'll get an output. Of course, we're going to do something a little bit more complex today, but this idea of node doesn't go anywhere as we start to stack more and more interesting pieces together. And the way we create nodes is we parse our data and we use node parsers. We go ahead and we take our documents, we chunk them, we eventually have node objects. Lastly, it's important to understand in Lama Index that the retriever and the query engine are really the things that this is where the magic happens in Lama Index. So what you want to think about is you want to think about sort of that retriever that helps fetch the most relevant nodes. And the query engine that is allowing us to sort of interface. These two pieces, you'll see a lot of retrievers and query engines in the code today. These are the two pieces that really are sort of the most interesting pieces to pay attention to as you start working with Lama Index. There's so much power in these retriever and query engines. We're consistently amazed as they come out with better and better ones all the time. The important thing about RAG is that it really relies on retrieval to be done well. And done well could mean a lot of different things in terms of evaluation. But what we wanna think about is we want to think about if we improve retrieval, then we improve generation. OK, and so getting retrieval right. Again, it's that data centric piece. It's so important. We talked in the last event that we did, High Performance RAG with Lama Index, about how to do a couple of things to go from simple to advanced. We talked about fine-tuning embeddings and we talked about using small to big retrieval. We talked about a sentence window retriever to do that. And this is one of the Llama Packs that have been recently put out. What we're going to talk about today is we're going to talk about how to go ahead and deal with embedded tables. And this is the embedded table sort of llama pack. This is a notoriously hard problem to date. Everybody asks about this. Everybody wants to see this capability rolled out in every tool out there on the market. Why though? Why is this so important? You know, it's kind of like the VP in your pocket problem. This is sort of the meta middle manager problem. The CEO is going to go to the VP of any given function or organization in the company or department, and they're going to say, hey, how's it going in your department or function? And in order to figure this out, there's a ton of words that need to be read, but there's also a lot of data that needs to be looked at. And maybe there's some summary documents that put together some of this context that we know is done on the analysis side, some context that was done just on the qualitative thinking through things and how to connect to strategy side. But a lot of this answering of these types of questions, how are things going in area X, area Y, area Z, these could really be streamlined if we could look at both qualitative and quantitative data at the same time very easily. And the simplest way to think about doing this is within one single document. So this is sort of the ground floor of this bigger overarching problem. Consider the annual report. This is a document people work very hard on each year for every public company out there. This is called a 10K filing, and it's submitted to the Securities and Exchange Commission, the SEC. This is a super important document, and it's a great one for us to think about today because it's got text, it's got tables, and it's got tables with data that really matters if we get it right. It doesn't have a lot of other crazy other things like super complex diagrams or electrical wiring diagrams or simply schematics of how to, let's say, put a piece of furniture together or to fix a piece of equipment. It doesn't have a lot of other pieces like actual images that people have taken and added to the document that you might see in technical reports or service manuals. It just has tables and text. And some of those tables are pretty complex in their own right. So this is a great use case for us to start with. Because if you're going to ask questions about an annual report as a stakeholder or as an investor, you want to make sure that you're getting the right answers back. So this is the use case we're going to look at today. How are we looking at this? How does it actually work? Well, the magic here where Llama Index has taken a tool from unstructured.io comes in the form of this embedded table, Llama Pack. And what it allows us to do is this hierarchical parsing. This idea of text splitting of tables. It's historically hard, but can we take that text data that's unstructured and the table data that's structured? Yes, we can in just five lines of code. So what we do is we start with a summarization of the table. It weization of the table. We check out the table, we create a summary of it, we're creating a schema for the table at the same time. This sort of structured table summarization allows us to put that together. Then we use hierarchical node references to sort of take that summary that was created and a link to the table and then plug that into a recursive retrieval process that is a lower level process in Lama Index and one that's been out there for quite some time. So we're sort of, again, as we solve more and more complicated problems, we're sort of stepping up layers of abstraction and layers of, you might think, metadata within our indices to allow us to solve these more complicated problems with constructs that we already have. The way we can visualize this is we can look at this unstructured element node parser that's going to be collecting nodes that are just text and also collecting nodes that are tables. You see the summary and schema here. This is what's being collected. Finally, this is fed, as I mentioned, into this end-to-end recursive retrieval. And what this recursive retrieval is doing is it's allowing us to explore the most relevant nodes and explore the relationships that these nodes have to additional retrievers and query engines. So at query time, if the index node is fetched, now we're sort of combining the idea of indexes and nodes into one construct called an index node, then the underlying query engine or retriever will be queried. So now we're sort of, again, stacking. One example of the way we might think about this and look at this is in this sort of contrived example, but one that allows us to see how this recursive retrieval is working. Tell me about the sports teams in Boston and the positive aspects of Seattle. The sub-question query engine is now breaking down sports teams in Boston into question 1 and positive aspects of Seattle into question two. The recursive retriever is then figuring out, okay, I'm going to go and send this question to this index node, which underlying it has a query engine and retriever that can be used, and the positive aspects of Seattle, of course, go to the Seattle index node. So this idea of sort of going to the right place for the right thing is all we're really talking about in this recursive retrieval process. So for today's build, what we wanna do is we wanna go ahead and grab some data. We're gonna get 10K filings from who better than to look at but NVIDIA. And as we get this thing ready to rock and roll, the first and most important piece is to process our data using all the constructs that we've seen so far. We're going to bring up the LLM wizard himself to show us how to get this done and get ready to build out the rest of our RAG system. Wiz, over to you, man. Hey, yeah. Okay. So hopefully my mic is a little bit better for you guys, but I'll just try to speak up if it's not. So the idea here is that we are going to be doing a conversion step. Now, there are some methods we can use that rely on OCR, but the actual performance of those tools was a bit lackluster compared to this. So this is the solution we're gonna use, which is basically we're gonna convert our PDFs into HTML files in order to be able to parse out the table information within. And definitely still not the best, but hopefully if I'm speaking a little bit closer to the mic here, I'm louder. If not, I'll just yell a bit more. So the idea is we are going to, yeah, we can definitely zoom in here. We're going to go ahead and we're going to add some additional information. First, we need our OpenAI key. And then we're going to add some additional information. First, we need our OpenAI key. And then we're going to grab this LamaPack. Now, the LamaPack that we're using is the Embedded Tables Unstructured Retrieval Pack. So this idea is that, you know, when Greg's talking about those five lines of code, this is one of the ones. five lines of code, this is one of the ones. So the idea is that we're going to use this pack in order to power our actual information. So you can see we just grab the pack from the hub and then we are going to store it in this directory. Now we have to do some data pre-processing. So we're going to use a very specific tool, which is going to be this PDF to HTML EX. If we look at the actual link here, it's actually a very good tool for this exact purpose. Very good at converting the actual information we want into the desired format. So how do we use it? Well, we just have to grab the package, and then we install the package, and then we can run the package as a subprocess in our Jupyter notebook. So we're going to convert this quarterly NVIDIA PDF, which you can see here, and we're going to convert that into an HTML file. And that's going to be what we use with our actual index and query engine. And all we need to do to set that up is actually call the embedded tables on structure retriever pack on that HTML and provided a path to save our nodes that we've processed. And that is it. That's the whole process to begin querying these structured, you know, documents. So let's at this point, go back to Greg, who will talk us through the next section. Yeah. Yeah, yeah, Chris. So, so cool to see exactly how easy that was. And, you know, just in case you're not tracking here completely, this is the embedded table functionality. And this is the process that as soon as we get that PDF into the right form, we're going to go ahead and be able to take that unstructured and structured data, do that structured table summarization, and connect that to hierarchical node references to do recursive retrieval. That connection to recursive retrieval is what we have to build out next. And that rest of our RAG process is what we want to focus on building out next. But the data piece, once we get it ready to rock and roll, you saw how easy it was. And so we really hope to see a lot of people start building, shipping, and sharing some amazing things with this new data processing. Because the rest of this is going to be pretty straightforward. If you've built rag systems with Thelma index before the models we're using today are models that you're going to be comfortable with. If you're building these tools at all today, open AI is GPT for open as Ada embeddings. Of course, you could plug in any open source embedding models or chat models here. And we're also going to be leveraging in terms of our tooling, we're going to be sticking with straight up Lama Index here. So we're not going to get fancy with the vector database, we're going to go ahead and create vector store using Lama Index. And we're going to go ahead and do the infrastructure. And if we wanted to do evaluation we could easily do that in llama index we're going to leave that out today but what we're going to do is we're going to essentially glue all the pieces of our rag system together with llama index we're going to put open ai models, and we're going to now leverage the data that we've chunked appropriately. We've created those nodes we need to do that hierarchical indexing that then we can retrieve in a more sophisticated way. So with that, I'm going to pass it back to Chris to show us how to do RAG for documents, including PDFs with tables the rest of the way. Chris? Yes, you bet. Okay, so I will continue to speak at a high value of our volume, sorry. And first of all, I just want to show kind of the output or the understanding of this uh specific uh tool so with the llama pack which is uh we're gonna walk through kind of step by step what that looks like underneath the hood uh we have the following uh you know uh output so we can ask questions that are simple like revenue and get responses that reflect the revenue uh we can ask questions that are simple, like revenue, and get responses that reflect the revenue. We can ask questions about revenue from sales outside of the US, which we're going to get a good answer for, because it's represented well in text. We can talk about any policy changes. You know, AI is going to change the actual policy, so that's fine. They are worried about it. That makes sense. When it comes to asking questions that are a bit more generic or hyper-specific, we might get a lot of noise, especially with the default retrieval pattern, which is going to be quite low. So we'll want to modify our LamaPack. Now, the LamaPack is just a loose abstraction of the actual code that we'll see in the next notebook. But the idea here is straightforward enough. We're going to load the path to our HTML file. We're going to parse that in an unstructured element node parser. And then we're going to build our index based off of this mapping that we can create, and then we're going to use our recursive retriever. Now, what we want to do in addition to this is perhaps modify the amount of retrieved documents or modify the LLM. We're going to do that by actually modifying the code that you see in, if we go to our files in our base.py, we can actually modify this code and then re-import it, and that's going to let us get a better understanding of, you know, this text, perhaps. So the idea is we're going to increase the amount of retrieved documents as well as we're going to move through a better quote-unquote model so that we should have better performance. So we'll just re-import the tool from our copy here, and then we will run it again. We can ask the same questions about policy changes. We get a better response because it's GPT-4 turbo. We can also ask questions that are quite specific to only table information. I mean, one of the things that we want to be able to confirm is that we can ask questions about only tabular data and receive an appropriate response. And indeed, we can ask the question about the research and development expenses and percentages of net revenue. And the only place that you can find this information is in the actual table. And it gives us the correct answer for both periods of time. So this is showcasing the ability to operate in that tabular only format. And that's the big idea. in that tabular only format and that's the the big idea. So how did this actually work? Well it's pretty straightforward and we mostly saw it in the other notebook. So we're going to first just again convert our PDF to HTML and then we're going to load it in using the flat reader from llama index. We're going to parse the nodes with our unstructured node parser this is uh using local inference so it's not going to the api we are using the raw nodes to get nodes from documents and then we're creating a map in this next step. Then we are going to go ahead and we are going to grab the actual vector index store, which is going to be constructed of those base nodes. Our retriever, which we're going to again increase the top k retrieved documents past the default one so we can collect more information which can be used to provide a better response. And then the actual you know query engine that we're going to use is this vector query engine which again we're going to use that similarity top k equals to three. We're going to use the recursive retriever from llama index in order to do this a little bit better. And then we are going to provide the mapping to our nodes as well as our vector retriever. And that's going to essentially be the whole process. From here, we can move on to asking questions about the PDF in the expected format. But the big idea here, and if we'll go back to the other notebook, the big idea here is a lot of this comes down to the efficacy or quality of this processing process, so this conversion process. Now, again, we tested with the OCR implementations and found them to be a little bit underwhelming. And this was the best format that we could come up with, uh, in terms of actually being able to answer questions about that data that's represented in our, uh, note or in our PDFs. That's not just, uh, you know, raw text, but that actually has some kind of structure or structured representation. but that actually has some kind of structure or structured representation. If you wanted to move to some kind of process that examined graphics or charts, you would want to move to a process that involved some amount of multimodal language model, but for the actual efficacy, this is the strongest solution that we found through the Lama Index tooling. And it does for the most part what it says. I think with a few adjustments at least, you know, actually modifying the model we're using, as well as increasing the amount of retrieved documents, we're able to get really good responses about the text representations of information as well as the tabular only representations of information where we're actually leveraging that structure in order to understand more about our data. And that is how you do RAG with unstructured and llama index on a complex PDF. That's it. Very cool, Chris. So, you know, it looks like that's a wrap for today. It looks like these HTM and HTML files, they work out of the box. PDFs take a little bit of massaging right now, but Llama Index and Unstructured do make it easier than ever to do RAG on complex PDFs and PDFs with tables. So, you know, get your hands on these tools, start building with them today, start making an impact at your company or in your startup or whatever it is you're trying to build next. These are certain to impress. And many more data types, of course, exist. Images, diagrams, all of this stuff is going to continue to roll out on the market. You'll be able to see us sort of testing each step of the way, but there's no reason to not try to start building with these tools today if this is a problem for your customers, for your stakeholders in your organization. And so with that, we're going to go ahead and move to questions. And I'd like to invite Chris back up on the stage to join me for the Q&A period. Chris. What's up? So, man, it did what it said. For the most part, yeah. I mean, we have to take this kind of loop-de-loose step of conversion. I see a lot of questions. I just want to address a few of them in the chat before we move to the Slido. So why are we not using the unstructured partition PDF is the big question, right? So because of the way that that's set up, it's actually not compatible currently with the tool stack that is existing in Lama Index. Even in the Lama Index demos that reference PDFs, they're actually just using HTM files. I think we're waiting on unstructured to get a bit more of a robust tool before moving on to actually using their PDF partition technology. And as for all the questions about privacy or the API, the API to unstructured is not being used. It's all being done locally. But the actual LLM is OpenAI and the embeddings are also OpenAI. So you would need to use locally hosted solutions to keep everything private, you could do so by again modifying that base.py file we saw to run your local pipeline. But outside of that, there's nothing else other than the OpenAI components that are going to be leveraged through an API or an external API, sorry, in this tool. All right, all right. So it looks like the first question that we have in the Slido. So everybody go ahead and please upvote your favorite questions in Slido. Is there a way to do hierarchical chunking on PDFs, such as document, section headings, subsection headings, paragraph chunks, this kind of thing? Yeah, definitely. I mean, it's a more engineer intensive thing to do. So it's not going to be like an out of the box tool, which is what we were examining today. You will have to build that and build those kind of like node graphs in terms of their hierarchy. But Lama Index does give you the tools to be able to do this. It just does take some work. So creating that metadata, creating those sections as you're parsing through your PDF files, and there you go. All right. Now, how important is the recursive retrieval? It seems a bit unclear, Islam asks. Can you sort of give your take on this, Chris? It's not super useful, but it is useful. Recursive retrieval is good because it lets us peer into deeper components of our node graph. So it would not work as well without it, especially the way that unstructured is creating these entities, right? So we're leveraging that unstructured node parser to build a fairly, not like super complex, but fairly complex representation of our data. And so we're going to be, the recursive retrieval is a good way to let us kind of dig deep into specific sections of our document as they have been parsed by our unstructured tool. Islam also asks, how much of an improvement is there using the HTML compared to directly parsing the notes from PDF? Yeah, like if you could use directly from PDF in an acceptable fashion right now that really preserved that structured information from your tables, I would say there would be information from your tables, I would say, you know, there would be not so much improvement. But right now, that doesn't work so good. So there's a huge improvement in going to the HTML representation first, and then, you know, building your index. Yeah. And that was one of the key things that we found as we investigated this tool, this time around is, you know know that that's an opportunity to reduce some friction potentially for you open source contributors out there you know if you want to jump in and help out with this i think that that's a really really nice piece to try to maybe bite off and start contributing to one of these projects on all right Then what are the differences, Chris, between Llama Index and Langchain, like the Retrieval QA chain? How do you look at these tools? And when should we pick up one or the other? Yeah, I mean, they're great tools. They're both great tools. Yeah, I mean, they're great tools. They're both great tools. You know, they're going to be able to do a lot of the same things, especially because they're both constantly implementing new and awesome things. And I mean, what it comes down to is just your preference and your comfortability with the APIs. They're all going to go ahead and eventually leverage something that's similar on the back end. We're constructing these representations. Out of the box, though, I would say Lama Index does take a lot of care with their concept of nodes and building these node graphs, I would say like Lama index for these more complex representation tasks is probably going to give you a smoother out of the box experience with less engineering hours than might require. Nice, nice, nice. Yeah. And, and, you know, this is a question that everybody wants to discuss all the time. We love sort of discussing these things within our community. If this is something you're interested in keeping up to date on, as the space rapidly changes, there's lots of things that one week, one tool might be slightly better at than another tool, and the next week we see it equalized again. So really, really interesting time to be in this space and to be asking questions like this. So Martin asks, do you believe that combining knowledge graphs with vector database retrieval provides substantial improvements, especially for big picture questions? Yeah. Yes. Knowledge graphs are a very rich way to store information. And with tools to navigate or traverse those graphs, it is, yeah, they are very good at those things. You know, if we can represent our information in a more information-rich way, then, you know, those KGs are going to be hugely advantageous for us. Definitely an interesting space to continue to look at. So tactical question, can this be used with outputs generated from AWS text track, text track table extraction? Yeah, I mean, if you can put it into a standard file format or any kind of, you know, usual representation, then yes, it can be used with this. If you have some kind of, you know, alternative method of converting the PDF to an HTML file or an HTM file that will also work fantastically. I saw a lot of people were talking about using Adobe's API, great solution. We didn't want to use an external API today, but there's a lot of really cool external, you know, managed PDF2X tools that exist out there. Next up, how do we get amazing PDF parsing without having to call unstructured.io's API? Their API, I think you already covered this. And it is ideal to do all the work locally. Can you be super clear about exactly what you did today? Yeah, I mean, it's just running locally. For the example that we went through today, that's the solution. You can run it through your API, but you can also do local inference. There's like a local inference package that you can import and leverage to do that without hitting an API. Now, this is run locally. And the next question is, would this solution actually work in a scalable cloud infrastructure? Yeah, I mean, it depends on what you mean. Yes, of course, it will work. It will scale like every other RAG application. It really comes down to what the different components you're using are, how you're leveraging them. But, I mean, this doesn't have any inherent scaling issues that are unique to it. So in the same way that it's difficult to scale complex RAG applications, generally, there's no new or not new challenges for this task. Yep, yep, yep. And so another couple of sort of maybe rapid fire questions here about how we can augment these systems. One, can this work with Lama 2, other open source models, or just OpenAI's GPT? It can work with everything. The model is agnostic to the rest of the stack. The model receives text information, though please do be sure if you're using an OpenAI functions query engine or anything like that, it will not be compatible. Query Engine or anything like that, it will not be compatible. And then on top of that, the performance is not going to be guaranteed to be at the level of say GPT-4 Turbo. If you're using like a 7B model, you know, with fine tuning, maybe you can get them up to par, but otherwise it's going to be a bit tough. Yep. Yep. Absolutely. And, you know, it reminds me, Yep, yep, absolutely. And, you know, it reminds me, if you're building these things, build with GPT, build with synthetic data, if your data is behind somewhere and you can't get it out in public, and then go ahead and check that baseline by changing to an open source model before you put your private data into that. We've got a couple of questions sort of jumping up in the slide out here. Anonymous, do you dump parts of the table for the LLM as context or the whole table? Any special format? We dump parts of the table as context with the unstructured tool. Yeah. Nice. Okay. So it looks like we've got some other movement here. Have you tried to implement RAG conversational memory and not just question answering? Yes. So you can basically just keep track of all of the text that your LLM has sent and your queries as a kind of conversational history object. All of the popular frameworks have like a specific memory object for this that you can use to maintain kind of a thread throughout the conversation. Nice. Okay. So if you were going to work with CSVs here and Excel files instead of tables within other files, would you use this tool or would you use a different approach? How would you go about that with CSVs? I would definitely use another approach. CSVs are already structured. They're structured in a understood and known way. So we would not need to use a tool like unstructured. We could use CSV loaders and CSV node parsers to achieve the same result. Or we could even take it a step further and do some kind of, you know, CSV, you know, tool that's going to let us actually calculate, you know, aggregate statistics about our CSVs or more. So, it's, you know, when we have these known file structured file types, we want to stick with using tools that are very performant with them. And we don't need to use generic solutions like what unstructured is doing. And we don't need to use generic solutions like what unstructured is doing. Okay. Okay. Ali asks, and maybe this is a point of clarity that we could add for everybody. Would it be better to convert the tables into a natural language description, like table to text explanation and create embeddings for them? Yeah. It depends on what you mean by better. But I mean, that's a great thing to do. I mean, the small part, we're doing parts of that by parsing through a table, developing some kind of metadata about it, right? Or describing what the table is about, what information it contains, and then using that to help us with our retrieval. But I mean, for the most part, if we care about the structure of the data, then we really want to keep the structure of the data. And in order to describe all that structure, you're talking about a lot of tokens, right? So I would say like in terms of like out of the box performance, it's going to work quicker, but it's not going to work as well. If you just say parse a table into some unit of text. Yeah. My sort of intuition was we're kind of doing that already in this tool. And then, you know, I was thinking of the example we had recently where we're looking at service manuals and trying to look up part numbers. And I'm just imagining sort of a part description part number table converted to text and I'm thinking is that really the best way to solve this problem so I think again it comes down to sort of it depends and yeah conserving that structure of the data might be very important lax asks any tips on scaling would say 10,000 documents how about a million documents? What do you think, Chris? Many, many, many, many documents. Yeah, I mean, the more documents you have, the more expensive it gets, the harder it is to retrieve the specifically desired information. And so the more complex your system needs to be in order to ensure you are actually retrieving what you need versus just the first thing that you find. You know, things like graphs can help us to do this. Things like really appropriate metadata application, filtering, and so on are also going to be key in helping us navigate these huge document spaces. But ultimately, it's just very expensive. It is, yeah, yeah. It's just very expensive. It is. Yeah. Yeah. It's just very expensive. And that's what it comes down to. Yeah. And, you know, in order to sort of try to decrease the cost as much as possible, in order to decide how to do that metadata filtering, et cetera, you really can start to dial down, okay, what are the questions I want to be able to ask? And then say, do I really need the 10,000 documents for that? If so, which parts of them do I need exactly? And you can start to, if you chunk it down into specific tasks you need to be able to do, which AKA specific questions you need to be able to answer, things you need to be able to get context for, that's going to allow you to do it in as efficient a manner as possible. Can you explain the node mapping process that you did? Basically, we're just constructing a kind of hierarchical node structure that we can navigate based on the kind of thing this is in a document. And that's about it. Okay. Mapping based on the kind of thing this is in the document. What do you mean by that exactly? The data type? Yeah, or what unstructured has classified this as being part of. So like, you know, spans versus paragraphs versus X versus Y. We want to understand the kind of text this is, the kind of, you know, document component that it is, so that we can better understand how it fits in with the rest of the data, what things are relevant to it or not relevant to it. Is it relevant at all to us? Right. So like a JavaScript header, we don't really need that stuff. Right. So all of this information is parsed out for us in a way that makes sense. But we need to map those nodes in order to understand what unstructured has done for us. Okay. Okay. Right. So as you start to get a little bit more into it, you may need to drill down into what some of these tools are doing exactly to decide on how things are being parsed. But yeah, I mean, if you're just applying it, let's see, go ahead. Can we get some out of the box results that are really good? So we got a question about PyPDF here. What do you think about PyPDF, Chris? Can it handle the PDF the same way as unstructured does? All of the PyPDF, you know, PyMu PDF, all of the PDF extraction tools are dope. And they do what they say they do, which is extract text from PDFs. The thing that they're falling behind a bit on is, you know, getting that structure out of the PDF. And that's why we've used the process we did today. But yeah, I mean, PyPDF is great. If you just want to rip text from pretty clear PDFs, I mean, that's going to be something that you can do well. To be clear, though, this is PDF file format, not scanned PDFs or images of PDFs. That's not going to fly. Yeah. Okay. So Charles is asking us to transition to talk a little bit about evaluating the quality slash, you know, how good this thing's doing. Is it hallucinating? Is it returning the right stuff? Am I getting good generations? How do you think about evaluating the quality of this particular system that we built today, Chris? Yeah, I mean, Lama Index has a lot of great out-of-the-box tools that you can use to evaluate. Not trying to just be like the Lama Index, you know, event today, but they do. They really do. Things like RAGIS are great tools to understand how your RAG pipelines are performing. to understand how your rag pipelines are performing. You know, it is, it's very, it's a very difficult problem because there's not like a perfect answer, especially for a complex task like this, where we want to be able to say like, okay, so what's our retrieval from the text of the document versus the tables of the document? Do we understand the table structure well enough? These kinds of metrics are harder to understand or harder to get information about and there's always gonna be this loss, right? If you do a conversion process from PDF there's going to be a loss. If you use OCR there's gonna be some loss, right? Like it's not gonna be perfect. And so we're always going to lose information when we're going from the PDF file format to a non-PDF file format, or even just extracting information directly from the PDF itself. And so we want to be building systems that are sensitive to that kind of loss. You know, things like do we lose structural information, I think is key. And using tools like RAGIS, LAMA Index, LM as a judge frameworks is going to be a great way to get that information and understand, you know, how good or how bad that system is. And more importantly, will give you directional signal when you make changes to see if you're getting better or worse. All right. I think the last question that we'll end on today, and feel free to riff on this for a little while, is are retrieval augmented generation and document question answering the same exactly, Chris? Is this a one-to-one that we should be thinking about here? No, kind of. So, I mean, it's kind of like one is a subset of the other one, right? So, document question answering is a subset of retrieval augmented generation, or at least we can think of it that way, right? We have documents. We are able to retrieve them and then make generations based on them. So this kind of idea of document answering is almost inherent, almost inherent in that RAG pipeline. It doesn't have to be, though. RAG can be used in ways that are exclusive to question answering. So I would say like they're, they're oftentimes used synonymously, but I would, I would more think of the, the retrieval augmented question answering or RACA as we like to call it is, is a subset of what actual RAG is and is doing. Yeah, I like how you brought that up. We teach retrieval augmented question answering as sort of the RAG 101 in our LLM operations course. And the reason is, is because this sort of subset idea, if you're gonna get started with something and you're getting into this field, it's very, very useful for you to think about RAG as simply a question answering machine. Although doing retrieval and augmenting generations is a more general process and one that we can use to dial in other things. If you go back to sort of classic NLP, question answering is just one task that we can do with a generative model. So you kind of get back to basics when you start to get into, is it just question answering? Well, no, but it's a useful lens through which to begin learning it. Well, Chris, I think we're going to go ahead and wrap up. That was an awesome Q&A. Thank you, Chris. Appreciate the demo today and all of your insights for Q&A. Thank you, everybody, for joining us today and for your participation. Thank you to Llama Index and to Unstructured for supporting the event and for letting us take a good hard look at those latest and greatest tools. This brings us to the end of today's event. And if it resonated with you and you'd love to build more cool stuff with us, Definitely check out our Discord, our community calendar, and our upcoming courses on LLM engineering and LLM operations from deep within large language models to actually deploying them in frameworks like Langchain, Lama Index, and cloud computing service providers at scale. We sort of cover as much as we can of the space to try to keep you up to date all the time. Join us on YouTube Live every week where we're investigating a latest and greatest tool, or we're diving deep into a classic principle that everybody should understand. If you have feedback on today, we'd love for you to share it with us. Please fill out the feedback form in the YouTube live chat. But until next time, keep building, shipping, and sharing, and we will most certainly be doing the same. Thank you so much, everybody. We'll see you all soon.
RAG for Complex PDFs
GPT-4 Summary: Discover the Future of AI Document Handling: Unveiling LlamaIndex and Unstructured.io's Breakthrough in RAG Systems! This must-watch event is perfect for AI engineers, business leaders, and LLM practitioners eager to explore cutting-edge tools for processing complex PDFs with tables, graphs, and diagrams. Learn how to integrate tabular and semantic data into RAG pipelines and enhance QA on intricate documents. Don't miss out on live code demos, insightful slides, and the chance to see if these innovative solutions live up to the hype. Join us to transform your approach to handling complex documents with AI!

Event page: https://lu.ma/RAG4PDF

Have a question for a speaker? Drop them here: 
https://app.sli.do/event/hGnryTXK1u6XMhnMG5ANKc

Speakers: 
Dr. Greg Loughnane, Founder & CEO AI Makerspace.
https://www.linkedin.com/in/greglough...

Chris Alexiuk, CTO AI Makerspace.
https://www.linkedin.com/in/csalexiuk/

Join our community to start building, shipping, and sharing with us today!
https://discord.gg/RzhvYvAwzA

Apply for one of our AI Engineering Courses today! 
https://www.aimakerspace.io/cohorts

How'd we do? Share your feedback and suggestions for future events.
https://forms.gle/WyH98AYHc6HxgL7PA