 Hi everyone, and welcome to Langchain, how to build chat GBT for your data. My name is Greg Lockman, and I'm the founder of the Machine Learning Makerspace, a brand new online learning community focused on empowering data scientists and machine learning engineers to build generative AI and LLM applications that create real value. We appreciate you taking the time to join us for the event today. Please share in the chat where you're tuning in from. We're so happy to see you at our first kickoff event. During our event today, you'll learn not only how to build a chat GPT-like interface on top of a single document using Langchain, but you'll also learn what it takes to build a multi-document question answering chatbot complete with agent memory and backup Google search. If that already makes you feel overwhelmed, don't be. We're going to take it step by step, one piece at a time, build everything up just like Lego blocks, and we're going to take it really easy when we get to the super advanced part. If you hear anything during our event that prompts a question, please follow the Slido link in the description box on the YouTube page. We will do our best to answer the most upvoted questions during the Q&A portion of today's event. Without further ado, I'm so excited to welcome my good friend and colleague Chris Oleksiuk to the stage today, as we'll be working as a team to deliver this lesson. Chris, what's up, man? Hello. Yes. How are you doing? Very excited, you know, to to be here. Yes. Yes. Chris is the founding machine learning engineer at Ox, an experienced online instructor, curriculum developer, and YouTube creator. He's always learning, building, shipping, and sharing his work like a legend. He embodies what we aim to build here at the Machine Learning Makerspace, and I couldn't be more excited to share the stage with him today. One quick note on the flow, I'll share concepts and Chris will do the demos. So with that, we're going to get right into it. the demos. So with that, we're going to get right into it. Welcome to today's event. Hey, Chris, what do you say we tell them a little bit about where we're headed with the data that we'll be chat GPT-ing today and share a sneak peek of the final product? Absolutely, Greg. Today, we are going to be heading down the rabbit hole. So we're going to be looking at some of the texts produced by Lewis Carroll. So the Alice in Wonderland series more specifically. So we're going to be using that as the documents that we're going to query across and chat with. And with that, we have an agent that helps us do that using land chain that we've named the Mad Hatter and so we'll just ask it a sample query here which is something like how can you avoid potential pitfalls of falling down a rabbit hole you can see that the system uses our agent as well as our supplemental chains in order to produce a response and that eventually will output a response to us unfortunately there's no specific information available how to avoid pitfalls of falling down a rabbit hole in Alice in Wonderland we know that our main character does fall down the rabbit hole so that makes sense many lessons to be learned and many things to ask the mad hatter from here um chris thanks so much for demoing that let's see how we built this thing up piece by piece so first off when we're talking about laying chain we're talking about chains chain, we're talking about chains. This is the fundamental abstraction innovation that we want to keep in mind at all times. And beyond that, we want to kind of build things up with core components of the laying chain framework. To do our single document question answering, we need a few pieces that are going to be really common for really anything that we build. We're going to need a model. We're going to need a prompt template. We're going to need to use a chain, maybe multiple chains, and we're going to need to create an index. When we get into multi-document question answering, when we get into agents, when we get into doing things that are a little bit more complex. We're going to add some additional layers, some additional pieces, but really fundamentally these same core pieces, these core components are going to remain just the same. So we're going to spend most of today's event focused on those. Most of today's event focused on those. First up, Langchain is all about combining LLMs with other sources of computation and knowledge to really create those complex applications. This is the purpose. This is what it's doing for the world today. This is what you should be thinking about having it do for you. So how can you leverage an LLM to create something even better with chains? A chain is nothing more than a sequence of calls to other components, including other chains. So it's a very generic abstraction, but it's also a very useful one. The single document question answering is going to essentially require three simple chains. And we'll see how these are built up in the following slides and demo. We're going to use the core components that we talked about in the outline, models, prompts, chains, and indexes. And we're going to take them one at a time. First off, we need a model. And the type of model that we're going to use is called a chat model. This chat style model is a little bit different than maybe just the way you've been interacting with LLMs in an input output text oriented way so far. Instead, the chat style, the I.O. schema that we're using is going to be an input of a list of messages, of chat messages, and the output is going to be a single chat message. Now, as we get into the chat model, we need to sort of differentiate between the types of messages that we're going to be using. The way that we do that is we define what are called roles. And we have the roles outlined here in yellow. We're seeing we have the system, the user, and the assistant roles. These are core, these are fundamental. This is directly from open AI, and this is how we leverage Langchain. It's thinking about these roles. So diving a little bit deeper into the roles, the system role is going to be that thing that provides a little bit of context, provides maybe a voice, maybe a user persona, maybe a character. We're sort of telling it a stance to take some place from which to answer the question. In this case, you are a helpful assistant is the system message. The user message is simply the user of the program. Could be you, could be anyone else using the application that you've built. And the assistant message is essentially the AI message. It allows us to act in place of the AI answering questions, effectively producing a one-shot example and a prompt input. Okay, so let's go a little deeper on that idea. After we just really note here that OpenAI and Langchain use very similar terminologies, but not exactly the same. So the system message is the system message, straightforward. User message in OpenAI is called human message in Langchain. And the assistant message is called the AI message in Langchain. Again, this allows us to provide outputs from the perspective of the AI that we are interacting with, essentially providing a few shot example. So let's check out how this works with some Kraft Mac and Cheese. Chris. Yes. Okay. So we'll pop into the code here. And I mean, first things first, we just want to be able to interact with our LLM using Langchain. So we're going to, in order to begin doing that, we're first going to have to grab some dependencies. So we're going to start with OpenAI and Langchain. Since again, like Greg said, we're going to be leveraging the OpenAI endpoint here. We just have to set up our OpenAI API key. So this is to ensure that we have access to the OpenAI endpoint. We have this helper function. This is just to display the text in a way that's nice to look at. It's not doing anything fancy though. And down here is where we first set up our chat model. Now, the first thing to keep in mind is that because we are using a chat model, so this is GBT 3.5 Turbo, we do have to make sure that we're using chat OpenAI from Langchain's chat models. And this is all we have to do to set up that chat model is just instantiate it with the model name as GBT 3.5 Turbo. Now we can leverage what Greg was talking about, right? So when it comes to the language that is used between Langchain and OpenAI, we have the system message, user message, and assistant message that Greg was talking to. In Langchain, that's the system message. And then the user message is instead referred to as the human message. And the assistant message is referred to as the AI message, but they're the same thing. So you don't have to worry too much about that. Just a naming convention. You'll notice that in our system message, we can input some text. So we have content, and it's you are a food critic. We have some content in our human message or our user message, which is do you think craft dinner constitutes fine dining? And then we have our assistant message or AI message in line chain with the content eGADS. No, it most certainly does not. So you can see that we've set up the system message, the user message, and the assistant message. We do this to guide how the LLM is going to respond when we give it a second user message, which is, what about Red Lobster? Surely that constitutes fine dining. We just need to combine these messages into a list. So we have our system message, our first user message, which was this, do you think Kraft Dinner constitutes fine dining? Our assistant message, which was the response that we're prompting it with, and then our second user message, which is meant to get a response from the assistant, or in this case, the OpenAI endpoint. We call that chat model that we built above with this list of prompts, and we can get a response. Ah, Red Lobster. Well, it may offer a casual dining experience with a seafood focus, I wouldn't classify it as fine dining. Fine dining typically involves a higher level of culinary craft. I mean, it goes on. The idea here is that we're able to provide all of this context or LLM in different and cool ways in order to guide how it's going to respond to us and how it behaves. So we'll go to Greg to learn about the next concept that we'll be leveraging. Awesome. Very cool, Chris. So Red Lobster, casual craft, not fine dining. Check. Single document QA prompts is what we need to do next. So we need to look a little bit closer at how we do prompting beyond the idea of this chat model. And let's sort of recall a couple of prompt engineering best practices. Many of you have done a lot of prompt engineering so far, and the core thing that we need to always keep in mind is give specific instructions. Now, beyond that, we want to provide some context. That's always providing some sort of role or voice or character, a place from which the AI can stand. Again, in this case, you are a helpful assistant. And then additionally, we want to really specify the input when we're doing prompting, right? Zero shot gives us some decent results sometimes, but when we can give it an example or two, that kind of gives us better and better results. So that's where we see in this OpenAI example, we're giving it one example output in the assistant role here. Who won the World Series in 2020? The Los Angeles Dodgers won the World Series. So we can ask it a follow-up question. All right. So as we get into sort of a prompt template, this allows us to do all of this a little bit easier and allows us to kind of do things over and over without replicating, copy pasting, prompt stuff all the time over and over again. So it's really a straightforward tool that you're going to have to use pretty much every time you go to try to build anything. In this case, we've got, you are an expert in subject and you're currently feeling mood and we can provide any sort of user prompt simply with this content here. So Chris, walk us through how this prompt template works and a quick example that we're going to do on sparkling water. You bet. So as Greg was discussing, one of the things that Langchain is best at, right, is reducing boilerplate. So it does this in a number of ways, both reducing boilerplate code, so code that you typically of ways. Both reducing boilerplate code, code that you typically have to write a lot of times, as well as reducing prompt boilerplate. In this case, we're using prompt templates in order to build, you know, kind of prebuilt prompts that we know are going to be effective for the task at hand, and then modifying them on the fly with specific user provided information so you can think of it like we're building an f string almost in python that we can uh get it set up properly by including these additional pieces of context so like we discussed you are an expert in subject and you're currently feeling mood. So the things between these curly braces are going to be replaced by user provided context. And the way we make that happen is by using this system message prompt template from templates. And then we provide this prompt template here. You'll notice as well that these do have roles. So this is the system message prompt template. So this is the system message prompt template and this is the human message prompt template. So you can have a prompt template for all of the different roles that your LLM has. We're going to build the human message prompt template to just be content. So it's going to be the user's question or query and that's all we're going to pass along to the LLM. In order to make this work in one shot, we need to make sure that we create chat prompt template from messages. These messages are these two templates we've already set up. So there's a system prompt template and a user prompt template. And this is familiar to what we saw above when we created this list, except this time we're able to format whatever we'd like in place of these variables. So let's see an example of doing that. We're going to use the format prompt method to format our subject to be sparkling waters, the mood to be joyful, and the content to be high. What are the finest sparkling waters? We're going to send this to messages just so we can send it directly to our LLM chat model. Now, all we're doing here is saying the subject becomes sparkling water. So you are an expert in sparkling waters and you're currently joyful. That's what we're doing. We send that to our chat model And we display it so it looks decent in Markdown And we get this response, right? So as an expert in sparkling waters I can assure you there are plenty of wonderful options to choose from We have Perrier, San Pellegrino, Topo Chico Gerolsteiner, and LaCroix And that's basically how we do it, right? We could substitute anything we wanted for that subject or mood, and we don't have to rewrite the prompt. We don't have to do anything like that. Lane Chain handles that for us through the use of the prompt template. We'll pass it back to Greg to explore the next concept we're gonna be leveraging. Nice. Very cool, Chris. Great to see how that prompt template can help us make decisions about not just what to eat, but what to drink as well. Let's see, as we get into actually chaining things together, this is no more complex than simply putting things together. So our LLM chain that we're going to use is the most popular chain that you're going to come across within Langchain. It's simply taking our chat model and taking our prompt template and linking those two things together with a code that really is as simple as what you just saw on screen. Chris, let's see how this works exactly. So on screen, Chris, let's see how this works exactly. You bet. Just as Greg said, this is very straightforward. We are just looking to chain our prompt into our LLM. So again, this is to reduce boilerplate, right? Sure, we could just wrap this in a function and call it whenever we needed it with the LLM, with the chat model. So just wrap this all. Instead though, we can with the LLM, with the chat model. So just wrap this all. Instead, though, we can build a LLM chain, which is going to have that prompt, have knowledge of that prompt and chain it into our LLM, as well as returning the response through our LLM. So in order to build this, all we have to do is provide our chat model, which we created in the first demonstration, and then our chat prompt, which we created just a moment ago, and put them into an LLM chain. Now we can call chain.run, which is going to run our chain, unsurprisingly. We can include our subject, our mood, and our content. So this time we're saying sparkling water again, just to stay on theme, but the mood is angry. And we're asking, is bubbly a good sparkling water again, just to stay on theme, but the mood is angry. And we're asking, is bubbly a good sparkling water? To which it responds, bubbly? Are you kidding me? That stuff is a disgrace to the world of sparkling water. There's nothing more than cheap imitation trying to ride the coattails of true sparkling water brands. The flavors are weak. The carbonation is lackluster. And don't even get me started on the aftertaste. So watered down disappointment. So this is the idea, right? We're able to modify these things on the fly and we're reducing the amount of boilerplate that we have to write when we're creating these applications. And while it's very straightforward with this simple example, right? Once it gets more complex, you're going to need to leverage these tools in order to effectively keep track of your prompts and how information is flowing through your application. So we'll pass it back to Greg to learn about the next concept we'll be leveraging. All right. So down the sparkling water rabbit hole, we found out that bubbly is disappointment and it can so moving on we've got indexing and this is really where a lot of the magic happens outside of just tapping into the llm when we're talking about building applications so we need a couple of different components here but this is where we get data centric this is where the data comes in and this is where we get data centric. This is where the data comes in, and this is where we put it into a form that the LLM can interact easily with. We're going to need things like document loaders, text splitters, text embedding models, vector stores, and retrievers. So let's try to break down some of this terminology before we look at the code to see if we can get sort of the big picture of exactly what we're doing. You know, when we're creating a question answering tool to look through documents, we essentially need to first create our index. And our index is our vector store. It is our vector database. In fact, a vector database is simply one type of index. It's the most common kind. So we're taking our documents, we're splitting them into chunks, could be one document, could be many documents. We're creating a ton of embeddings, which is essentially turning those words, in the case of the documents we're looking at today, into numbers and into vectors. looking at today into numbers and into vectors. Then we're storing those vectors in the Vector Store. Simple enough. The Retriever allows us to then search that Vector Store. It's an interface that allows us to query the Vector Store and really get a response of what is most similar to what it is that we're looking for that exists inside of our data. And that's the question answering chain is kind of the index and the retriever store. So let's double click in a little bit on index here. An index is a generic term, so don't be scared of it. It's simply a way to structure documents so that an LLM can interact. But really the index that you probably care about today is the vector database, aka the vector store. This is just what, you know, we just kind of talked about where it's numbers, the vectors are being stored. kind of talked about where it's numbers the vectors are being stored and the lang chain default is to use chroma db that's the one that we are going to use today and we will share a link with you in the chat a little bit more on why lang chain chose chroma db as the default lang chain supports a ton of other vector databases and that that's something that we're gonna get into in future events and in future community content. But for today, we're gonna focus on ChromaDB, the simplest possible index, which is a vector store. And as we build up a single vector store, sort of the canonical vector store steps are we're going to load documents. We're going to split up the text. Now, splitting of the text is more of a black art than a science. So Chris will kind of walk us through that. We're going to create embeddings from the text. We're going to use kind of the industry standard today. The OpenAI Ada embeddings model is what we're going to use in our application. And then we're going to store the vectors. This vector store is kind of the backbone of the retriever, which simply wraps around the vector store and allows us to query in natural language something that we're looking for. And it looks for something similar inside to take out and it does this really really fast that's the benefit really of the vector store and timely that it does it really really fast and not surprisingly, we're going to ask it about why the rabbit is running so late as Alice chases him down the rabbit hole. Chris, let's see how this works to put all this together. Yes. Okay. So first things first, we have to get our document in, right? So in order to query across documents, we have to have some documents. We're just going to go ahead and W get one, which is going to be the first Alice in Wonderland book by Lewis Carroll. She's just going to name Alice underscore one dot text. Essentially, first things first, every time is you have to get the data to Python. We're going to load this into memory using just classic Python here. Nothing lame, Chan, about this. Once we have it loaded, we can go ahead and start thinking about splitting it. There's a number of ways we can split text. And we kind of need to split text, in most cases, anyway. Because LLMs don't truly have infinite context windows. So you can't shove everything into the context window of an LLM. So the Character Text Splitter is gonna help us break our data down into bite-sized chunks that retain as much information as possible, but are otherwise just there to ensure we can grab only the most relevant context we don't need a bunch of other context as well if we include too much context it could potentially confuse the llm and so we want to avoid that as well we're just going to use the character text splitter here we're going to go ahead and we're going to split on the new line character and we're going to split on the new line character. And we're going to have a chunk overlap of zero, as well as a chunk size of 1000. Now, what this means is that we're going to every time there's a new line, we're going to potentially split if what's after the new line is too much for our context window. So you can imagine we're just splitting apart by new lines until we have, you know, the most information in this 1000 character length window. All we have to do is call dot split text on Alice in Wonderland. And it goes ahead and splits it, we get 152 resulting chunks. So this process is also called chunking once we've finished chunking we can go ahead and create our open AI embeddings model now this is as easy as just pulling it from blank chain we have embeddings.openai and then we get the open AI embeddings we're just going to use that Ada embeddings model from open AI's endpoint as as Greg was discussing. We have to get a few dependencies since we are going to be using ChromaDB. And as well, we're going to be using TickToken in order to tokenize correctly and make sure we have the right number of tokens when we're actually running this through our embedding process. In order to embed, though, all we have to do is call Chroma from our vector stores. We use dot from texts on it. We pass our text. That's our 143 chunks. We pass the embeddings model. And we just add some metadata. This metadata is just associating each passage with its number in the 143 long sequence. passage with its number in the 143 long sequence. And then we of course have our retriever, which is the retriever wrapper as Greg discussed, so that we can query this in order to get relevant documents from our vector store. Again, we're storing everything as numbers. So any information that flows into this will be embedded. And then it will be the relevant text will be extracted. So you're going to be able to interface with it through text. We can see an example of this by asking it what is the rabbit late for? We use the get relevant documents method on our retriever. And we just pass in our query. And we can see that it finds some context that's, you know, relating to the rabbit being late says, Oh, the Duchess, the Duchess, won't she be savage if I've kept her waiting, right? So we know that the Dutch is going to be mad if that rabbit is late. Finally, we're able to integrate this into a QA chain, Um, finally, we're able to integrate this into a QA chain, which is again, built using an LLM and we're using this chain type of stuff, which just means we're going to stuff all of the relevant context we found into the prompt. Um, so that our LLM can leverage it as potential context. And then we're going to pass our query. We're going to run our input documents, which is our docs extracted from get relevant documents. And that's going to be about it. We can call this chain using chain.run. And we know that the rabbit was late for something, but it's not specified what he was late for in the given context. So this is basically putting all the steps we discussed earlier together. We have our chain, we have our prompt, we have our retriever, and finally, we get our response. So we'll pass it back to Greg to continue on with learning. Yeah. Thanks, Chris. And it's really interesting to see how we kind of have to be real specific we have to sort of learn exactly which context to use uh how exactly to interact with the data how we're chunking the data matters uh each piece of how we build up this application really does affect the user experience. And so it's really, really interesting to see all of this kind of come together. This is, in an image, what we just built, this single document QA. So we enter a query. We use our templated prompt. It looks for the answer within the vector store that we created with ChromaDB, that we took our documents and we chunked and converted to embeddings before we put in there. And the interactions with the LLM occur on the search and as well as when we're doing our prompt. So this is sort of the simplest, one of the simplest, most common things that you can do, the single document QA, and this fundamentally is a natural language interface for your data, chat GPT for your data, right here in an image. And so what we wanna do today is we want to give you some insights into how to build a much more complex application, but we're not gonna go through each piece of code. We are gonna share the code, but we're not gonna go through every single piece of it because of time constraints on the event today. However, if you do have questions, please do share them in the Slido and we will get to them at the end, or you can wait till the end to ask them in Slido as well. We should be able to answer any and all questions that you have today. So with that, we're gonna take it to the next level here. And again, we're getting more advanced. We're gonna to do a multi-document QA with Google search chat bot. All right. So we're going to add, and we're going to share the Colab notebook with you now, although we're not going to go very deep into it. Chris is going to produce a Colab notebook run through post event that we will share. Okay, so this is a little bit more complicated, but again, we have the fundamental chains, the prompt chain, the tool chain, the data indexing chain. We just have a few extra pieces in each. So what are those extra pieces? Well, before we get to that, let's talk about agents for a second. This is one of the most confused concepts within LLMs and within generative AI right now. And I think one of the best ways to think about it is with a quote that I pulled from a book that I was reading recently called Complexity. Agents is sort of a generic term. Agents might be molecules or neurons or species or consumers or even corporations. Molecules form cells. Neurons form brains. Species form ecosystems. Consumers and corporations form economies and so on. At each level, new emergent structures form and engage in new emergent behaviors. Complexity, in other words, is a science of emergence. And what are we doing today? We're using Langchain to build complex LLM applications. Agents are a key way that we can take our complexity to the next level. Agents in Langchain are simply in a similar way to indexes. Langchain talks about these. They are used for applications that require a more flexible chain of calls to LLMs and other tools. Agents essentially have a tool belt. They have access to a suite of tools. You can think of Agent Smith from the Matrix with a tool belt, although it's not a perfect analogy. Which one to use in the tool belt is based on the user's input, and the output of one tool can be the input to another. There are two types of agents in Langchain. We're going to focus on action agents to build our Mad Hatter agent chatbot today. Now, the action agent simply receives input, decides which tool to use. In this case, we're either going to leverage a search of our index, or we're going to do a Google search. And it calls tools and records outputs. It decides the next step to take based on the history of the tools, the tool inputs, and the observations. Now, this history piece is what requires us to add a little bit more to our tool chain, specifically the memory buffer. We have to remember what we were doing so that we can select the next best step. In addition to the memory buffer, we're also adding Google search to the tool chain. And in addition to everything else, we're adding multiple types of documents. So this is different file types and multiple documents to our data indexing chain. But again, fundamentally, we're going to use data indexing chain, a tool chain, and a prompt chain. And that's really the key components that we need to add. There is some additional complexity when we go to implement the code. But what we want to show today is we want to show how this all comes together when we build not just in a Google Colab notebook, but when we actually create a chainlet application, a chatbot-like interface, a true chat GPT for your data interface on top of this multi-question answering agent system that can also do Google search. So Chris, can you walk us through a few examples? So maybe we can learn not just what the Cheshire cat is up to, but maybe if we can interface a little bit with that agent to learn how it's working without digging too deep into the code for today's presentation yeah of course yes i mean really quickly i'm just going to go through a couple of the concepts that we're adding so we added more data it's basically the same process we did before but you add new data uh we're persisting that vector store it's great the first real thing we're going to leverage in order to get that chat GPT-like experience is we're going to add memory to our chain. Now, we're not going to talk too much specifically about this, but the idea is that we can provide both a conversation buffer memory, which is like what's happened in the conversation so far memory, as well as read only shared memory, which is this idea that some tools can get access to the memory, but they can't actually change it, right? So conversation buffer memory can be modified. This read only shared memory cannot. So this is useful for tools that should be able to leverage memory, but shouldn't be able to commit anything to memory. In order to add that to our chain, all we have to do is include it. Memory equals read-only memory. You'll love to see it. We're going to set up a couple of tools. You can think of tools as an extension of chains that can be leveraged by our agent that sits on top and gets to see the descriptions and choose which tool is right for which job. In this case, we have our main QA system, which is our index powered QA retrieval chain. And then we have a backup, which is Googling, right? So obviously this is not going to be every case. You're not going to just be able to Google. But if you can, this is an example of that. We create the actual agent. This is just showing you how you can go through this. We don't have to focus on much here, except that we have full control over how the agent acts, what it's supposed to do. We have the ability to give it this chat history so that it can make decisions based on its memory. We have the ability to ask it questions, which are input. And then we have the ability to let it quote unquote think. You can leverage that thinking a little bit more complexly, but for right now, we're just going to let it do what it needs to. We create our zero-shot agent with our tools and then with our prompt templates, and then we provide it what the inputs should be. We create our chat model. This is going to be our LLM chain that kind of powers that agent. And then we set up our zero shot agent with that LLM chain. This is the same LLM chain that we had before with our chat model and our prompt, our prompt being the agent prompt. And then we make an agent executor? Basically all this is is it's making something that's allowed to call other tools, call other chains, and then use those outputs to strategize or come up with a more clear or concise answer. Once we have all of this set up, we're able to ask it the question, what is the deal with the Cheshire Cat? We can see that it enters the new chain. It has this thought. This is the agent executor. It has this thought, and it decides it needs to use the Alice in Wonderland QA system, which is our main retrieval QA system. We're going to ask that system, what is the deal with the Cheshire Cat? It returns a response, and our executor makes a observation, which is that Cheshire Cat. It returns a response and our executor makes an observation, which is that Cheshire Cat is a character in Alice in Wonderland. It gives an answer. Then the thought of the agent is that it knows the final answer. And so it could just give us that when we see it here. The Cheshire Cat is a mischievous and enigmatic character in Alice's Adventures. It is known for its distinctive grin. I lost my scroll bar. I'm so sorry. see it here the cheshire cat is a mischievous and enigmatic character in alice alice's adventures is known for its distinctive grin i lost my scroll i'm so sorry for its instinctive grin and ability to disappear and reappear at will so then i asked the question well what makes it enigmatic and this is where that chad gpt experience comes in right we don't have to ask what makes the cheshire cat enigmatic we ask what makes it enigmatic and our agent knows right i'm not sure about the specific details that makeeshire cat enigmatic. We ask what makes it enigmatic. And our agent knows, right? I'm not sure about the specific details that make the Cheshire cat enigmatic. So it asks, again, the QA index, and it gets a response. The response is the Cheshire cat is enigmatic because of its ability to disappear, its riddles, and its knowledge of Wonderland. So then I ask, well, what are some of those riddles? The Cheshire cat, you know, he has riddles, but we want to know that we ask the context in the context doesn't find it. This is likely because riddles isn't present. He probably doesn't say I'm gonna ask you a riddle now, Alice, right? He just asks a riddle. So we have to go to our fallback tool, which is the Google search, we can see that the agent doesn't find any riddles in the qa retrieval train. And so it goes to the backup and it Googles what the riddle is. And we get the example of a riddle, which is what road do I take? Now, again, what is Alice's response to that riddle? We're not having to provide that context because of the memory. We're in an ongoing conversation here, right? So in this case, because it knows what that riddle is, it's what road do I take? We can actually see the agent executor decide to ask our QA retrieval chain, what is Alice's response to the Cheshire Cat's riddle? What road do I take? And because it's provided the riddle itself, we get the context and the context provides us the answer, which is I don't care. I don't much care where, so long as I get somewhere, which is Alice's response to the Cheshire Cats riddle. So with memory, with the agent executor, with the tools, we're able to have a seamless chat experience, just as you expect from GPT 3.5 that includes our context that we've given this particular agent through the QA retrieval chain. And with that, I'm going to pass it back to Greg to wrap us up. Boom. That was awesome, Chris. That was so cool to see how exactly the chatbot is working behind the scenes, what it's thinking thinking how it's making decisions um really just a lot to take in there as soon as we start playing around with agents um really enjoyed you walking us through that from today's event you know we really hope that we showed you that the potential for building truly complex applications is there, you know, really emergent applications. It's all there. It's just like, what are you going to build? Single document QA is really a fantastic entry point for anybody getting started with this tech. It is going to teach you about the foundational constructs within the link chain framework, and it's going to allow you to get that chat gpt like experience with your data doing simple queries although if you want to take it to the next level after you master those basics and you want to get that true chat bot chat gpt experience you really can't beat adding agents adding additional ways of finding information, and really just getting creative with what exactly the pieces are that you're chaining together. So, you know, thank you so much for, you know, Chris, for showing us that. Thank you everybody for joining us. We've got the Google Colab links. We're going to share the slides. The chainlit demo that you saw today as well in the beginning, we're going to share that. Reach out to us directly with any questions that we don't answer today. Greg at mlmaker.space or Chris at mlmaker.space. And with that, I'd like Chris, join me back on stage as we go through the questions of the day. So if you have additional questions, please go ahead and add them to the Slido. We've only got a few going today. So if there are more questions, we've got time for them. Otherwise, we'll go ahead and break early. All right. So first off, love this. Thank you, Anonymous. However, am I correct in assuming there's no way to, quote, ground these models without sending data to the OpenAI endpoint? I'm concerned about PII, et cetera. Yeah. I mean, so you can, luckily. Open source models generally are performant enough now that you can use them to power this, which means you can run them on-prem. You can also use your own proprietary models. You can also use Azure services to provide you like a closed source OpenAI instance in which you don't have to expose any PII. So there are many different ways that you can kind of control the PII flow that doesn't expose it to any public endpoints or semi-closed endpoints. So you can ensure that your customer's information is kept private and never leaves whatever, you know, contract contract data contracts you have it never leaves that scope so um you you are unfortunately incorrect but also fortunately incorrect right so uh you don't have to use uh open ai but for this example specifically we are leveraging it yeah we're hearing that more and more aren't we people trying to move away from open ai but um it's a great teaching tool i feel like we find uh it's a great entry point play around with some open source data with it um but yeah if you're using your own data certainly uh might be worth looking at some other things um okay so uh from not a number we have if raw tabular data with labels is presented as a as a text document and QA is like quote predict new label for row where row has new feature values would this work sure I uh why not uh yeah I mean you can make it work right so we can format the response from our LLM through Langchain using format or response formatting tools that are provided within the Langchain framework. So you can format it right back into the row if you want. There is a lot of research and papers that indicate that the LLM can be used in such a way. So to predict like what the next thing is going to be, it's obviously not going to be without some serious modification or introducing new tools. It's not going to be like great necessarily, but if you've built say a custom LLM that does this, right? And you hook it up to Langchain, it can absolutely be leveraged as a tool in that the agent could use right so you can put it into that flow uh but i would say it depends on what performance you need or what you're using it for but you can do that yeah absolutely yeah right like almost like if you can dream it you can do it with this stuff right right? I mean, it's generic as it gets. So the next question is like the question that we keep getting like everywhere we go, Chris, like what open source models do you recommend for on-prem, et cetera, et cetera? Yeah, I don't. Unfortunately, and fortunately, the models do as it exists right now for lightweight LLMs, or open source LLMs, or however you want to define what you're meaning by open source here. It they're all specialized in some way at this point, or can be specialized. And so I think that when it comes to different tasks or different part of the agent chain, there's different responses to this. A more general, well-performing Instructune model is likely going to be your best bet for like the agent, right? Because that has to be a general, it has to have general knowledge and ensure certain response response format and then within each of the tools you know there's models that are better at that kind of open QA right uh or close QA I mean it's it's totally dependent I would say though if I have to give an answer start with something like open LMS open llama or Falcon 7B these models are doing well enough to kind of fit wherever you want in the stack. You do have to keep in mind that it's going to error sometimes, right? So this is a, not a deterministic task. Sometimes it can give a response that doesn't make sense. And so you're gonna have to build some custom error handling or parsing in order to kind of get over that hurdle that you might not have to if you use a much bigger model, right? So Falcon 40B or OpenAI endpoint. But the idea is if you need an answer to just get started, Falcon 7B or OpenLM's OpenLama are a great place to start, as well asify's new model is also fantastic if you were looking to uh to use that great insights and and i think the other layer of this is commercial availability versus no commercial availability so definitely check the license on the models that you're looking at if you're just playing around and learning it doesn't really matter but as you start to do things for your company for your business this is another key question that you're looking at. If you're just playing around and learning, it doesn't really matter. But as you start to do things for your company, for your business, this is another key question that we're getting all the time. So that's definitely a deep dive for another day, but a definite emerging space there, which model for which application, which size, and which commercial availability, how much privacy, all of these things are really interesting open questions. All right, so from anonymous, can we ask this bot to summarize the information from our knowledge base in a particular way? Like, can we set the system persona, for example, as a product manager? Yes. Yeah. You sure can. I mean, it's not an exciting answer. Yes. Yeah. Yeah. I'm kind of thinking like, how would you train it to be more like your product managers? I don't know. You can kind of fall down the rabbit hole a little bit there, but yeah, you could certainly just kind of in that system message that we saw within the chat model, kind of just, you are a product manager and start there. Also I did misspeak, I'm very sorry. Not Shopify, Salesforce, the new Salesforce. Yes, okay. Good call, good call. Okay, tactical question. How do you control the size of the tokens sent to the model by Langchain? There's a max tokens parameter that the OpenAI endpoint accepts. And so you can set it. You can also limit it on your side using a number of Langchain. So like that max tokens is for the response by the model. If you're wanting to limit what's sent to the model, you can also just use lane chain to that. There are parameters that you can set that help you to do that. You can build custom functions to help you organize or determine how much tokens you're willing to send. All of those things can be explored by lane chain on either side of the LLM chain. Yeah. As a follow-up to that, great presentation. How would you go about utilizing Langchain to create an application to generate docs that are three plus pages long for commercial use? Yeah. At that point, we have of think about what are LLMs good at. We have to think about what's the best tool for the job. If you're wanting to do it all in one shot with very good context adherence, you're looking at some of the larger context model, context window models. This is something like Anthropix, Mpt or even now open AI has like up to 16 K tokens for chat GPT 3.5 Turbo so there's a number of considerations to make another way is to break it down into parts only do one kind of you know part at a time so paragraph by paragraph I mean there's a number of ways you can approach this, but ultimately what it comes down to is context window and how much do you need persistent context through the entire three plus page long document. I would stick with your kind of clods and your big context window GPT-4s if you really wanted that to shine because they are the best at those long contexts right now. Yeah. And finally, we have our last question from Deepak. Other than the Langchain website, is there any other website you recommend to learn Langchain? Also, in your experience, what is the best way to make retrievers? And I guess I'll just throw out the deep learning AI course that they recently put up with Harrison is a great place to start. Gives you some insight, some overview. I'd recommend taking the prompt, the chat GBT prompt engineering for developers first to kind of get the vibe and feel before you head into that. But you could have those two done by the end of the day today they're just a couple hours each maybe one to two hours um any other thoughts on that Chris and then on retrievers I gotta I gotta do my catchphrase thing here whatever so I would recommend just building with the tool to learn land chain just build stuff with it like you're gonna find cool stuff in courses and like you're gonna find cool stuff on websites but if you don't have a reason that you that you're building or you don't have like a thing you're excited to build you just like they're not gonna stick like they would if you're trying to solve a problem i would also say like really don't worry about the fact that you're going to have over solved a lot of the problems like this is potentially overkill for a lot of tasks uh right you could you could do these with much simpler models and everything like that but just use the tool to solve a problem um you know even if it's over engineering like it gets you used to the tool gets you in it and then the best way to make retrievers is there is some trial and error involved, really understanding your data, understanding how to chunk it, understanding what potential context you could be losing, and, you know, setting those overlaps, setting the kind of chain you're going to use and the kind of retrieval you're going to use, right? So like, we have normal cosine similarity, but maybe MMR is better, because that takes into account what we've already retrieved. So we're expanding our potential context space. There's a lot of little fiddly knobs you can use, but I would say the best way is to really understand your data and then leverage that information you have to set the correct parameters and use the correct chains. set the correct parameters and use the correct chains. In other words, build, build, build. Okay, awesome, Chris. That brings us to the end of today's event. Thank you everyone for your participation. This has been brought to you today by the Machine Learning Makerspace. Our community is just getting started and we're so grateful that you joined us for today's event. We're also excited to announce that our first ever four week live course, which is going to be on LLM Ops, LLMs in production, is going to be offered starting August 14th. That's going to be the kickoff date. So in the follow up email that you receive after today's event, we'll share not only a post-event survey, but all the details on our upcoming LLM Ops course. Chris is also going to put together a long-form video explanation of that multi-document QA chatbot with agent memory and Google search, and we'll be sure that you receive that as well. For everything else, please follow Machine Learning Makerspace on Twitter, on LinkedIn and YouTube, at ML Makerspace on Twitter, Machine Learning Makerspace on LinkedIn and YouTube to stay up to date on everything that comes next. Until then, we'll keep building, shipping and sharing. We hope to see you do the same. Until next time, everybody. Later.
LangChain: Build ChatGPT for Your Data
GPT-4 Event Summary: Dive into the Future of AI with Our LangChain Workshop: Build Your Own ChatGPT! Discover the cutting-edge Large Language Model Operations (LLM Ops) and master LangChain to create sophisticated LLM applications. This interactive session will unveil the secrets of using chains to integrate prompts with LLMs and vector databases. Learn to craft advanced QA chatbots using just PDF documents, with all demo code available via GitHub. Ideal for anyone eager to harness LangChain for personalized LLM applications or delve into the world of LLM Ops. Don't miss this opportunity to shape the future of AI!

Event Page: https://lu.ma/r3q7ndj4

Have a question for a speaker? Drop them here: 
https://app.sli.do/event/bbCdYRphpHe9KTjfz8S4Ek

Speakers: 
Dr. Greg Loughnane, Founder & CEO, AI Makerspace
https://www.linkedin.com/in/greglough...

Chris Alexiuk, CTO, AI Makerspace
https://www.linkedin.com/in/csalexiuk/

Apply to our upcoming LLM Ops: LLMs in production course on Maven today! 
https://maven.com/aimakerspace/llmops

How'd we do? Share your feedback and suggestions for future events. 
https://forms.gle/WgviJKqdL9ZsqnFEA