 Hey everyone, today we talk LangSmith, the end-to-end production LLM application tool leading the way in LLM ops. In this event, you'll learn how to prototype and evaluate a RAG system with Langchain, providing a baseline for improvement. From there, we'll see how Lang chain and Lang Smith can be used to improve LLM applications in production. We'll see lots of features, lots of tools, we'll cover the core ideas. And we're going to go ahead and really get into it soon today. My name is Greg, and I'm founder and CEO of AI Makerspace. Of course, I'm joined by my man, Chris, the LLM wizard and CTO. He'll be up shortly as we've got a couple demos for you today. Thanks for taking the time to join us. If you've got questions along the way, go ahead and get those into Slido and upvote your favorites. We'll have time for Q&A at the end of the event. All right, let's go ahead and jump right in. We've got an hour together, and to learn Langsmith, we're learning basically everything all at once. So let's align our aim for the day. And as we always do, by the end of this session, you will understand how to baseline RAG applications with evaluation and end-to-end testing. You'll see how to improve RAG applications through quantitative metrics and through leveraging human feedback, developer feedback, and annotations within LangSmith. We'll get sort of a broad overview of exactly what the space looks like and exactly what we can do today, sort of at the edge of production LLM applications. We're going to break this up into two key areas. The first one is more of that AI engineering, that prototyping. And as we dig into that, we'll realize that once we have something deployed to production, it's now time to do a little searching and researching, do a little data science to take it to the next level. This is where we believe the data scientist of the future is really going to have a lot of important work to do. Once we get those applications into production, then it's time to optimize them. Then it's time to get data centric and look at the data science. We're going to focus first on AI engineering and what lies at the center of these infrastructure tools like Langchain. What they're doing fundamentally is they're helping us to combine LLM applications with other sources of computation and knowledge. We've seen Langchain sort of emerge over the past six months, since their series A over the past eight months, and really start to dial in exactly what they're all about. We saw just a few months back, they're talking about being data aware, connecting to other sources of data and knowledge, as we've heard, and moving more towards these agentic reasoning applications as well. More recently, we see that they move from the sort of data-aware language to this context-aware language. In the age of RAG, this makes a lot of sense. And then simplifying to sort of talking, talk about reasoning, machines, and applications, this is another core tenant of what langchain is all about today interestingly if you joined us last week we walked through this diagram to teach you about lang serve today we've seen some updates just in the last week we've seen some key updates to this diagram including the playground including talking about now this lang chain core in lcel the lang chain expression language and also looking at the community piece which we assume is going to be the open source pieces that remain free and open in lang chain we also see this nice cognitive reasoning cognitive architectures sort of templates and lang chain. We're interested to see how that continues to evolve. So hopefully you'll get a feel for the big picture of this today. And we've got lots of resources to help you dig into the details. So we're gonna move quickly through each of the core component pieces to make sure we can get to Langsmith today. LCEL is the core language that allows us to build and deploy both in LangServe and in LangSmith. It really connects well to the tracing and monitoring functionality that can help us with debugging in terms of LangSmith. That's going to be our key focus for today. And when we talk about models and prompt templates and really engaging in a chat style way with a chat style model, we want to make sure that we understand that we're setting up these templates to leverage that system, user, and assistant message schema that is sort of one-to-one when we go to system human and ai message in langchain we've talked about this a lot before we're going to keep it simple today using openai and langchain and it should be pretty straightforward exactly what we're doing as we set up our models and chat templates of course we're doing a rag system, so we need to make sure that we're building in each of these component pieces along the way. We're not going to really deep dive each of these pieces that we have as we have in the past, but we are going to remind you that RAG is all about finding references, adding those references to the prompt, improving your generations based on the references that you find. So when we ask a question, we go and we search a vector database. We set up our template so that we can take the stuff we find in the vector database that's similar to our query, stuff it into the context of the prompt, thus improving our generations. This is the basics of retrieval, and we will be setting this up as we get into our baseline. Of course, we need to chain things together. That's what laying chain is all about. And this is sort of the layer of abstraction that comes after we get the nuts and bolts and the core components really set up and ready to rock and roll. The chain is kind of the thing. It's just, it's nothing crazy. It's just allowing us to connect pieces to other pieces, models to retrieval systems, et cetera. And so this Lang chain expression language, this really elegant piece down here, this is what we're going to be using. This is what's going to allow us to integrate very nicely into Langsmith. And of course, we're going to be chaining things together along the way. What we're going to build today is we're going to actually go and find some rich content from Langchain's blog. We're going to build a rag flow specifically out of Langchain's blog. We're going to search Langchain for the top resources for any given question. We're going to be able to ask specific questions, and we're going to be able to get really nice generations and fact-checkable answers with references. We're going to keep it simple today in terms of models, and it's going to become clear why. But OpenAI is a great starting point for learning. And since we're learning something awfully complex, let's not add additional complexity today. We've also opted to leverage OpenAI's GPT 3.5 Turbo as our chat model today. Excuse me. And that's for good reason that we'll talk about in just one second as we get into our rag system we want to note we are using the the langchain blog and we are leveraging facebook ai similarity search to build our vector store out that's an open source tool and finally to get this baseline set up we need to talk a little bit about evaluation. This evaluation process, this sort of feedback loop, allows us to then not just glue this whole rag system together, but start to really ask the question, how can we improve this? Should we change our chunking strategy? Should we try different retrieval methods? Should we add some fine tuning? Should we try different retrieval methods? Should we add some fine tuning? Should we augment our data in some way? We can start to ask really poignant questions to improve specific aspects of the performance. And as we talk about evaluation, it's important to understand that in a RAG system, we have the question we ask, we have the predicted answer, the result here in this slide, we have the context, the returned we ask. We have the predicted answer, the result here in this slide. We have the context, the returned references, and we have the ground truth called the answer in this slide. Generally, that ground truth is best created by humans, but so often these days, we're doing the ground truth generations with the most powerful model we can find out there. Shout out to GPT-4, GPT-4 Turbo today. And since we're using GPT-4 as the ground truth generator, we're going to go ahead and leverage that GPT-3.5 for our predicted answers or results. In Langchain, we're going to leverage a evaluation metric called chain of thought QA, and it's very simple in the way that it works. The prompt template is very instructive here. You are an expert professor specialized in grading students' answers to questions. You are grading the following question, query. Here's the real answer, answer, ground truth. You're grading the following predicted answer, result. This is the prediction. Respond with correct or incorrect. We can use the LLM in this way to evaluate things automatically. Yes, evaluation is always going to come at some sort of cost. And we are going to take this to the next level in terms of the chain of thought QA, where we're going to not just instruct that you are an expert professor, but further instruct to use chain of thought reasoning. All right. Now we're ready to baseline everything. We're ready to see it rock and roll in Langsmith. We've got the Langchain blog. We've got the RAG system. We're going to see it rock and roll in lang smith we've got the lang chain blog we've got the rag system we're going to do it with face and open ai and all this comes together in our first demo i'm going to pass this one off to chris the wizard to show us how to get this up and running baseline so we can take it to the next level soon. Chris, over to you, man. Hey, Greg. Thanks so much. Yes. So I will go ahead and walk you through. First, I'm going to walk through kind of the notebook that we have set up here. And then I'm going to walk through actual the Langsmith interface. So the idea is that we have this idea of, you know, two separate entities that are working here. One is our co-lab, which is going to represent our application. This is what could be running in production or anywhere that it could be running. And then we have, of course, our Langsmith backend, kind of, you can think of it, which is where we're going to send traces to. And that's what's going to be what powers our tool. So first, we're going to grab our dependencies. We're going to use our OpenAI API key to generate an API key. And this is the idea here. That's, you know, we're going to use OpenAI for basically everything today. So we're going to set it up there. Next, we have to set up a basic rag chain. We're going to use a GBT 3.5 turbo, as Greg described earlier, for the reason of we're going to need to ensure that we are able to mark these responses or evaluate them correctly or responsibly. And so we're going to use GPT-3.5 Turbo as our generator for our application and the GPT-4 as our evaluator. You'll notice that we've got this tags thing here. This tags thing is pretty interesting because it is what's going to let us kind of, you know, set what resources we're using in LangSmith so that we can have better line of sight to what, you know, particular resources being used when, and we have a way to reference that for when things change, right? So these tags are very powerful part of the application. And we'll have the notebook link for you very shortly. And I'll drop it in the chat when we do. So the async bug handling, this is just necessary. We have to do this in order to do this in Colab. So this is like a processing step. Then we have a sitemap loader, which we're going to use to load the Langchain blogs. So Langchain's blog very, you know, happily has a sitemap loader or a very robust sitemap. We're going to use that to ensure that we get all of these blogs. So there's 155 total blogs. So we're going to load those up and then we can see that we have these sources to point us back to the original blogs, which is dope. We're going to use a recursive character text splitter. So we're just going to use a very naive splitting strategy here. There's no particular rhyme or reason for it. We know that these blogs are going to be relatively parsable text. So we're just going to use a recursive character text splitter. relatively parsable text. So we're just going to use a recursive character text splitter. This is one of the things that we could modify or play with and then determine the results through Langsmith with, which is pretty awesome. We wind up with a total of 1100 documents, which is not so bad. We now need to embed all those documents and store them into a vector database. So we're going to go ahead and do that using the OpenAI Embeddings tool. We're gonna be using TextEmbeddingAta02 in order to do this properly. We're gonna use a face vector store. So face is like Greg said, just meta. It's a great, quick, lightweight vector store. We're gonna go ahead and load up all of our documents into there and embed them. And then we're going to set it and load up all of our documents into there and embed them. And then we're going to set it up as a retriever. So now we have a retriever. We have a generator. We need to create a prompt. So we're going to create our base rag prompt here. The basic idea here is that we just want to say, using the provided context, please answer these questions. If you don't know the answer, say you don't know. Then we're going to pass in context and a question. And then we can set up our LCEL chain finally. So we're going to use LCEL because it's well integrated with the rest of the Langsmith and Langchain ecosystem. It has a number of production benefits, but it also has a number of benefits when used with Langsmith. And so that's what we're going to do today. Our base rag chain is our... I've added some annotations here in the notebook so you can look through it. But the idea is that we're going to retrieve our question from our input. So if you'll notice, we're calling this with this dictionary question. So we're going to grab our question question. So we're gonna grab our question and that's gonna become our question for the next step. And then we're gonna grab some context, which we're gonna get by passing our question into our base retriever. So you can see that we now have this dictionary object, which is gonna have keys, context and question. Context is gonna be filled with context from our base retriever based on our question. And then question is just going to be that question pulled forward. We're going to do this next little piece of the puzzle. This is just so we can pass that context forward so we can retrieve the context through the output. So we want to be able to look at our retrieved context from our application. Now, in our last step, straightforwardly, we pass that, you know, context in question into our base rag prompt, which is going to format the prompt here based on the context in question. And then we're going to pass that to our LLM. And then we're going to pass that into our string output parser and then we're going to retrieve our context going forward and that's it so let's test this out we can ask a question what is a good way to evaluate agents we get a response based on the blog and then we also get a bunch of context which is going to be a list of documents the idea here is that we're going to have both a response and some context which is what we need in idea here is that we're going to have both a response and some context, which is what we need in order to ensure that we're well grounded, right? So we're grounding our responses with that context. So we want to retrieve the context. So now we have that set up, it's time to finally get into Langsmith, right? We've got our chain. Our chain's working great. We're loving our chain. And then we're going to use our Langsmith environment variables to start this process. This is all you need to do. If all you want is simple monitoring, you could stop right here, which is the power of Langsmith. So we're just going to set Langchain tracing v2 to true. We're going to name our project. We're going to point to the Lang Smith endpoint. And that's all we have left is to actually get our Lang chain API key, which we're going to get from the user. You can put this into an environment variable, set it as your secrets, whatever service you're using. That's all we have to do. With that, we can run this particular chain, and we can notice that in this chain, we've given it the tag demo run, right? So let's go into Langsmith and see what this looks like. So you'll notice that in Langsmith, we have a bunch of different, I'll zoom in here so you can see it a little bit better. We have a bunch of different runs, right? We have a bunch of different, you know, expansions that we can do. We can click on them, but we want to look at that specific demo run that we tagged. And the way that we do that is we can go to our tags and we can check demo run. And then we can view the actual run that we just did in the tracing tool. So you'll notice that this tracing tool is, I mean, it's pretty powerful, honestly. It is an incredibly robust trace. We can expand this all the way out and you see all the steps involved in using Langchain. You can check to see which steps took the most time or the least time by clicking on this run stats button. We can see how many tokens we used. When we hover it, we can see how many propped versus completion tokens we used. You know, we can see the total time for this chain to run. This is the power of Langsmith, right? We're getting all of this kind of for free, right? We are, we're getting this all just from setting up those environment variables and then running our actual command. That's it, right? So we have access to this whole trace out. We can see every step, what every step did. We can see our retriever steps. We can see our prompt steps, our parser steps. Everything is available to us, which is absolutely incredible. So now we're able to see kind of some of the benefits, right, of Langsmith already. You'll also notice that we have the ability to see our latencies. So this is giving us an idea of how long it's taking for our users to run their chains. We can also see if we look at this, you know, we have every call that's made we can look at just the llm calls that are made if we wanted to we can see a monitor of kind of you know the different call counts that we have we're just doing this over one day so we're not seeing a huge uh difference but if we look over one hour we can see you know we have these LLM calls happening, our success rate, so we can monitor health. We have our average latencies, P50, P95. We have so much information, and all we did, right, to get this is set up these environment variables, okay? But Langsmith can do so much more, so we're going to look at how much more in a second. But for right now, we're just going to look at how we would run a simple evaluation, right? Greg said we got to baseline this. So let's baseline it. The first thing we're going to do is create a mock data set. Now, you can create this data set however you want it. You can create it with a human expert, with AI experts. It's totally up to you. It's entirely in your hands what you're going to be able to use this for. So in this case, we're just going to go ahead and we're going to produce a number of questions manually. And then we are going to set up our Langchain client. We're going to set up our dataset name. We're going to create the dataset in Langchain. And then we're going to create the data set in Langchain. And then we're going to add examples. So we create an empty data set. And then we're going to add individually examples based on these questions. What we're providing for the answer in this case is actually just the context that our retriever pipeline retrieved. So there's a number of different ways you could provide an answer. You could have AI answer the question. You could have GBT4 do it. You could have anything you want. In this case, for the chain of thought example, we're going to go ahead and provide these contexts. And those contexts are going to be what lets us have the ability to do our evaluation. Is this a correct response or not? So let's look at what that data set looks like right here. You can see that we have our data set. We can see our examples. We can click on each of our examples where we have our inputs and our outputs. And we can see all the linked runs associated with those examples. You can also see that we have the ability to run tests. And we'll see tests right in this window. And we'll have access to exactly how these tests shook out in the code or in the evaluation. This is a very powerful tool. Extremely good at what it does. And what we're going to do is we're going to look at the first baseline that we got, which is going to be this contextual accuracy of 0.83, helpfulness of 0.67, and a litness of 0.83. We're going to come back to that metrics in a second. But for right now, this is the idea. We're doing pretty good, but we could be doing better. So we'll kick it back to Greg, who's going to introduce to us how we might do it better. And then we'll see how we do it in the code. Yes, Chris, there's so much in Langsmith for us to try to get a handle on that. It's really kind of crazy. And, you know, trying to really wrap all this together in one it really truly is an end-to-end production platform as you guys can see you know just getting rag up and running and over to langsmith and the ability to trace it just kind of gets us started and this is really the point of productionG applications and production LLM applications is that once we get them into an environment where we can actually do some improvement, it's time to really do some data science. It's time to do some LLM ops. It's time to really try to fine tune this thing in many different ways to really dial it in for our application. You know, we've seen that this new age of the AI engineer is kind of upon us, where it doesn't take long. It doesn't take five months. It doesn't take five weeks. It doesn't even take five days to prototype anymore. It takes just a few hours to get a prototype up and running and into production. But then what happens once you get there? Well, that's when things start to get really interesting. And that's really where the edge of all this stuff is today. For instance, once it's there, what are we evaluating and why? We just saw Chris outline correctness and harmfulness and custom metrics like litness. We can literally, sky's the limit with eval for us. And that's not even talking about really focusing in on super specific retrieval metrics and trying to enhance our baseline retrieval and therefore get better generations. Of course, we can do fine tuning. If we have specialized language, we might want to fine tune embedding models. We might want to continue to fine tune down to the specific task our users are using our application for with the chat model that we pick up off the shelf. So it's really interesting to sort of talk about going from basic to advanced RAG. There's lots of ways to do this. Of course, the black art of chunking is always there. There's various ways to look at different indices that you can create and have different metadata filters and reasoning machines, sort of agentic thinking on top of all of those. And really, there's so many interesting things that we can do. We're going to keep it super simple today. In the interest of time and introduction, we're just going to do a simple re-ranking on our retrieval system, and we're going to see how the metrics can actually go up. So if you are using Langchain, picking up an ensemble retriever is something you might want to go ahead and just try off the bat. This is the idea of really trying to take a bunch of references and then order rank them appropriately. So you can think of sort of a hundred references. How do I just take the top five references? What does it mean to be the best reference exactly? Well, it's something like not a ton of redundant information, it's something like directly correlates to the ground truth answer. And there's many different ways to think about evaluation today, but're going to go ahead and pick up the hybrid retrieval ensemble approach to just show you one quick example of ways to enhance rag that's going to allow us to take a closer look also at and open up some time for us to take a closer look at annotation and feedback. You kind of got some of that from Chris's demo already, but here's the big idea, right? Is in order to align these things with not just humans, but specifically our users, our customers, people who are using our product, that's who we really want to align these things with. And feedback can be as simple as like thumbs up, thumbs down, RLHF status, but it can also be much more complex. You imagine that the data scientist of the future almost has to be a UX person as well, almost has to really be engaging the user for those additional pieces of feedback, especially early in the product development lifecycle, where you're really getting this dialed in with early adopters and visionaries. You as the developer are going to do this on your own in Langsmith at first. And this is where some of these feedback tools and annotation tools come in. Because although you can sort of get that thumbs up, thumbs down, oftentimes you want to just sort of make notes that you can come back to later and you can think about as you continue to curate data sets for the future. So for instance, if you're seeing something that isn't quite right, doesn't quite align, you may want to go back and you may want to modify that data set, modify that vector database full of your information in some way to really enhance the retrieval process. And yes, probably improve some metrics, but qualitatively make the output better for your users. Because remember, as we build LM applications, we want to prompt engineer as far as we can. We want to get it into deployment as quick as possible. Often we're moving from prompt engineering to question answering, AKA RAG to fine tuning. moving from prompt engineering to question answering, AKA RAG, to fine tuning. And as we're sort of providing annotations and feedback on the way people are interacting with our application, we can start to imagine ways to improve our prompting. We can improve our instructions. Whether it's zero shot, we can give additional examples of specific things that might show up in the future. One shot, two shot, few shot. And of course, as we get more and more and more and more and more examples that we're collecting along the way, we might realize that actually the way people are using this isn't in sort of a generic LLM way. It's solving one super specific task that we didn't really see from the outset. And so as we move along this sort of dimension of more and more examples, we ultimately get back to fine tuning. And so we can control for uncertainty and optimize that user experience through prompt engineering. We can use metrics like harmfulness and helpfulness directly built in to Lang Smith now. And of course, we can use that reinforcement learning type of approach directly as developers or working with our users to make those guardrails even better. As we think about the model behavior, we want it to be more and more constrained over time more and more dialed in over time more and more useful for that specific thing we're doing with our users and we want it to be grounded in the right data this annotation and feedback is going to allow us to do both it's going to allow us to optimize in both dimensions both towards optimizing the llm and the way the model acts, as well as towards optimizing the context and the way we're grounding the model with our retrieved information. Because it is a meandering road to optimize these specific types of applications, right? We'll prompt, we'll use one shot, two shot, few shot. We might do some rag, we might do some fine tuning. From there, what do we do? Maybe we pick up an ensemble retriever. Maybe we start annotating and noticing real specific patterns. How can we get to that human level performance so that we can feel comfortable deploying this at scale? We're going to have to potentially work with our users. The tools in Lang Smith allow us to streamline this process more than ever. And of course, ultimately, if you're seeing the same thing over and over, maybe you want to continue to annotate, provide a little feedback on that. Hey, this is the same or similar prompt semantically to something I've seen in the past. Start collecting those, start caching those prompts, start saving money as this thing rolls out to more and more users, more and more inference, more and more cache if you don't cache. And then as we get to fine tuning, we're talking about cheaper models, running inference on more quick, easy to deal with models. And that's just going to be better for business overall. So it's all about improving the system once we're in production. And to sort of show exactly how this works, we're going to go back to Chris for this improving RAG demo, where he's going to do a deeper dive on some of the things we've already talked about today and show exactly how you can take your RAG system to the next level chris back to you man heck yeah thank you greg okay so we're gonna touch on a couple things here first of all we're gonna touch on re-ranking so what is re-ranking how do we do it why do we care well re-ranking is basically a way to cast a wide net and then, you know, you know, pare it down. So the idea is that with reranking, we're going to be able to, you know, take a large set of potentially relevant documents and then trim it down to the most likely relevant documents. So we're going to use Cohheres re-ranking to do this. We're also going to test this kind of claim made by Cohere, improve surface performance with a single line of code. We're going to see if this is really true. And so we're going to first set up our base retriever. We're going to use the same retriever we did before. So this is our face vector store. The only thing we're going to change is that we're going to increase to 10 relevant documents. So instead of having the base three or two, we're just going to use 10, which is pretty big. And then we're going to use this re-ranking tool to re-rank those results into the most relevant results. So that's the idea here, right? We're going to spend more resources, computationally speaking, in order to do this, right? So in order to improve our results. Now, we're going to recreate our chain using exactly the same chain we did before. The only thing we've changed is we're using the re-rank retriever. I removed the annotations. So if it's easier for you to understand or interpret, that should be helpful. And the idea is that we have this tag where we're going to include a tag on this chain called cohere re-rank so we can keep it straight in our heads and in the code and in Langsmith. And now we're going to do this, you know, this kind of more robust evaluation set. So you'll notice we're still doing this chain of thought QA to check the veracity of our answers. We're still doing this harmfulness detection, but we've also added these few different labeled criterias. The first one is helpfulness, which is going to determine if it's helpfulness, take into account the correct reference answer. Then we're going to see if it is lit. Is this submission lit, dope, or cool? You know, this is obviously like a playful example, but the idea is that you can create these custom metrics. You can create them in even more robust fashion if you wanted to. The idea is you can create your own custom metrics. You can create them in even more robust fashion if you wanted to. The idea is you can create your own custom metrics that help you understand things about the responses in a way that might be more aligned with how a human might think about if a response is good or bad. Again, this is a playful kind of thing, but it is important. You'll notice that in all of these, I have this prediction key response, right? So we see this in literally every one of these. This is because my chain will return two distinct objects. My chain will return both a response and a context. response and a context. So when we want to have our criteria, we want to compare that to whatever base answers we have. We need to key it to a particular response. And so that's what we're going to use a response for. We can get very fancy with this and use it to compare our contexts and all kinds of other things with custom evaluation metrics. But for now, the idea is we have to address our actual response. But the way to do that is fairly straightforward, which is dope. We're going to also have our three additional metrics. These are out of the box from Langchain, which is conciseness, coherence, and relevance. These are going to help us understand how concise, coherent, and relevant our responses are to the particular user's query. And now we can run the eval. So first of all, we have our kind of base chain in which we have this ability to understand, you know, we have a decent accuracy, no harmfulness, which is great. We have some okay helpfulness, some, you know, some middling conciseness and litness and all this other stuff. Okay, that's nice. That's great. And then how does the re-ranker perform in the same thing, right? So we have the same set of results and we find that our, for the most part, are, they're well aligned. They're within the same kind of range. We get a little bit better on helpfulness for our re-ranker. And we can look at this in our data sets as well. And we can look at the actual responses. We can see here that our contextual accuracy is 0.83. Our coherence is very high. We have no harmfulness. We have some helpfulness. And litness is okay. We can actually compare this to runs that we did with the other tool, with the other chain. So we can do a side by side comparison of them in order to determine the quality or reasonability of the response. So we see here for the most part that these are well aligned, but our chain in terms of the re-ranker is actually quite a bit more lit. And you'll notice that it's uh a much more lit chain you know with very little uh improvements made to the chain thanks to coherence re-ranker so their claim makes search better uh you know with a single line of code if you're talking about litness i mean surely we've shown that right here but the idea is that we can look at these results and we can think about them in terms of, you know, what, what are we failing on? Now, one of the things that you might notice is that the latency is a little bit higher on our, uh, our actual chain with the, uh, re-ranker. And so we can look at our responses, right? We can kind of use our responses to determine how or why that happened. And we can look at the whole chain as a whole to see what happened here, right? Or where did this fall down? How did this, you know, how did this work out to be so, so much more high latency than some of the other responses, right? So if we look at this, like, 24 total thing here, we can see that we have, you know, this, this particular evaluation took quite a bit longer. Okay, that's dope. If we look at the, you know, these lower kind of instances, we can see that, you know, this was incredibly quick. And we have the ability to debug and trace through everything that's in our LankSmith application very straightforwardly, right? And this is what's going to be what lets us take this kind of to the next level. So say we wanted to address this particular run, right? So we look at the results. We say, okay, it's coherent. You know, it's not harmful. It took 6.5 seconds though. Maybe that's a bit long. Oh, this one took 10.74 seconds. Let's look at this one, right? Let's open this run. And we can see when we click on the show run stats, we can kind of see which points took the most amount of time. And we see that the generation here, right, took 7.24 seconds, which is quite long. So it's not our re-ranker. Our re-ranker is only taking, you know, 1.23 seconds. It's not so bad. It's definitely a little bit slower than it would be without the re-ranker, but it's not so bad. We can see that actually this generation took a long time. And now we can kind of look and see why. We see, oh, this is a very verbose kind of response. Okay. We can also see, you know, this is using our base LLM. So this is the tag we set up in our notebook earlier, if we look towards the top. So this is using our base LLM. Took a long time for the base LLM to get that back. So maybe we could look at, you know, some of the metadata that we have access to. That's great. Okay. What if we wanted, though, a better way to, quote, unquote, you know, deal with this? What if we wanted to be able to, you know, take this run and then learn something from it and improve it, potentially, right? And the way that we can do that is we can actually use our annotation queues to do that. Now, our annotation queues are going to be something we can use to metricize or understand our responses outside of, I'm just going to zoom out a little bit. The text might be a little bit less readable, I'm very sorry. But just to make this fit the screen better. But you can see here, like, we can use this annotate queue to determine a lot about our responses from, like, a human perspective, right? So not only can we look at the automated evaluations, but we can also use this annotation queue to have human annotators help us understand our data, right? So, labelers, data scientists, whoever is going to wind up doing this job, we have the ability to do a whole bunch of things. First of all, we can rate if this is a correct response, right? So, we can look at our actual response, which is our output. We can read it through, look at our query, and we can say, yes, that is correct, actually, right? And then we can say, oh, you know what? We're going to add some feedback. So we're going to add litness. We're going to say this response is about a 0.5 out of 1 litness score. And then we can add a note, right? Let's say we were looking through this and we wanted to look at the run. We can see like, dang, that took 10 seconds, right? And the LLM took how much of it? The LLM took 9.96 seconds on its own. So we could add a note saying LLM generation took, you know, about 10 seconds might be worth looking at. And then we can add all this feedback, and then we can mark this as done, and we can move on to the next step. So how do we add things to an annotation queue, right? Like, that seems very powerful. Well, the way we do it is we can create a new queue. Let's just call this one correctness, you know, check to verify result is correct. And then we can create that. And now we have the correctness annotation queue. If we go back to our project, what we can do now is we can select runs. And then we can add them to an annotation queue. And it's as easy as that, right? Now we go to our annotation queue, we can see we have this check to verify if the result is correct. And this is all available through Langsmith, right? We didn't have to do anything to get this functionality. It's just provided for us, which is an incredibly powerful tool. We're going to look at one more thing before I kick you guys back to Greg, and that's going to be the playground, right? So let's look at that. Let's go back to our kind of, you know, our beefy 10-second example, right? We're going to open this run, okay? We're going to go to our LLM, which we know took the most time because we were just looking at it. Now that we have our LLM open, I'm just going to zoom out so you can see the button a little bit clearer. We have this playground button, right? So what we can do is we can click the playground button, and this is going to get us a playground that is interacting with the open AI model that we have selected, right? So the idea is that we have the ability to now troubleshoot potentially why this took so long. You know, we can say things like this, please keep your responses short and concise. keep your responses short and concise. And then we can see if this is going to be, you know, thanks Grammarly. Grammarly is telling me to make an adjustment here so we can see this. Okay, this looks good. We can add additional messages like system messages, user messages, or more. Let's just see how this works. We would need to provide our OpenAI API key, which I will do off screen so you guys don't wind up seeing it in the video or the recording. But the idea is here that we're going to, and then we're going to hit start. We're now going to send that to the LLM and we're going to see we get a much shorter response, right? 2.47 seconds versus that last seven seconds. And so this is how you can leverage the tool of Langchain to really optimize and improve your LLM based applications. There's a number of different tools that we can use that help us give us visibility to what we're doing. And on top of it, give us access to troubleshooting and evaluating. We can also see that even though this is a new run, that run was not stored in our Langsmith project. It was created in its own playground project. Super helpful for keeping these two things separate. And with that, I'm going to go ahead and I'll kick it back to Greg. Yeah, Chris, man, that, there's so much that we can do in Langsmith. And, you know, it's great to see how we can dial in to reduce latency. We can specify an evaluation metric we want to really go after and hit it as hard as we can. And we can just use it for, it's almost like it's, it's almost too much that it can actually do. So super awesome to see all the ways that we can improve just a simple RAG system. And it's really starting to become clear. And hopefully to some of the audience today how improving RAG systems is really going to be part of hopefully to some of the audience today, how improving RAG systems is really gonna be part of this data science of the future as we go into 2024. What we saw today is we saw that LangServe is sort of leading the way in solving these end-to-end LM application in production issues that so many people are talking about, but there doesn't seem to be one tool to bring everything together. This streamlining of improvements, it spans the entire spectrum of prototyping, because once we're in production, we might want to improve prompt engineering, we might want to improve the specific aspects of RAG, whether it be retrieval or generation. And of course, we may want to eventually move towards fine-tuning our LLMR embedding model, but that baseline is so key. If the baseline is 10 seconds or if the baseline is 0.5 correctness, we want to make sure that we can move those numbers up or down depending on our targets. So interesting to see how LLMs act as evaluators through simple prompting, whether we're prompting them for correctness or a made-up metric, like how dope it is or how lit something is. This baseline evaluation is going to be highly dependent on your scenario and what you value in your company or for your application. So we're really looking forward to seeing what you guys build, ship, and share with Langsmith. It is still in beta. So as it comes out and as it gets rolled out, more and more people will start using it. Today, you can sign up to get access to it just like Chris has and start using it today to take your LM applications to the next level though. And with that, we're going to go ahead and go to Q&A. I'd love if you guys would go ahead and kind of let us know what questions you do have. It looks like we're light on questions today. And that's probably because we covered so much, Chris, in such a short period of time, bit of a tornado, bit of a whirlwind. But if you do have questions, we're here to answer them for the next 10 minutes or so. It looks like we've got one question in the chat so far. So go ahead and throw it into Slido or even throw it into the YouTube live chat today if you want to talk more about your specific issue. But Chris, what is your experience? What's our experience today at Makerspace with other languages? Like for instance, Spanish. We a spanish speaker asking about this question yeah so i don't have any specific line of sight to spanish uh i have some to uh so written chinese um with the open uh or the uh zero one ai ye models which are fully bilingual um when it comes to like interfacing with the tool, it is English, and it appears to be English only. I could be mistaken, and I'm very sorry if I missed some localization options, but right now it appears to be built in English. But it's going to be able to do the same things that it does now with the corresponding LLMs. So if you have a multilingual LLM, it's not like Langsmith won't work with it. But the interface is going to be for the most part in English. And then the actual quality of bilingual models or multilingual models right now leaves a lot to be desired, especially when it comes to this kind of meta stack that we're developing on top of, you know, our LLMs, which is like LLM as a judge, right? LLM as X. That doesn't necessarily translate as well to other languages. And so you wind up in a situation where you're translating a lot or you just don't have the functionality. So I would say that's kind of like a ecosystem issue, not necessarily a, it doesn't really have anything to do with Lang Smith or Lang chain, but definitely an ecosystem issue. Yeah. Yeah. Yeah. And I, I am noticing a lot more Chinese and Chinese English models coming out capabilities coming out it seems like that's a really big area of of development today that is important to keep an eye on I haven't seen sort of Spanish or or any of the languages spoken in India really emerge yet in the same way but definitely going to be interesting to keep an eye on that. So Cameron asks from the YouTube chat here, any advice on building a benchmark data set? Yep. Pay humans to do it. If you want the best one, golden data set, I think is like the accepted term that we're using these days. A golden data set is worth its relative weight in gold. You know, getting experts to annotate and develop really good data sets for evaluation and benchmarking is going to be your best bet. Second best bet is things like if the domain is compatible, you know, LLM or synthetically created data sets are pretty good. They're definitely much cheaper and much easier to produce. And so maybe that volume might counteract, you know, the expert cost. But if you want like the best data set, humans are still the go-to. Synthetic data, though, it's coming up fast. And I mean, there are papers that kind of read in this way that suggest it is as good, but the benchmarks are all over the place. Yeah, yeah. Humans still the GOAT. Okay, yeah, love it, love it. That's what we like to hear here at AI Maker Space, yeah. So I want to go ahead and go to Vinesh Iyer, who asks, how does this compare to like 1B and TrueAir's TrueLens? Are they along the same space trying to solve the same problem? Or is there something fundamentally different here? I mean, the only fundamental difference is the deep integration with Langchain for Langsmith. Otherwise, they are trying to solve similar problems, right? This idea of visibility monitoring and now moving towards the observability space, moving right this idea of visibility monitoring and now moving towards the observability space moving towards this idea of actionable intelligence or intelligence that you can automatically action so it is the case that they're solving similar things or similar problems in similar ways they do have their. If you're in the Langchain ecosystem, you have access to Langsmith. I would recommend using the tool. In terms of the rest of it, it's just going to be a preference thing and which features line up best for you. So Langsmith's evaluation is pretty clean and simple to use. 1B, Phoenix Arise kind of tooling, the evaluation is less out of the box, less canned. And so that might be a benefit for you. But you can integrate like Ragus with any of them or whatever Stanford renamed Ragus to when they did their recent paper, you can use that as well. But the idea is you're solving the same problem in pretty similar ways. Yeah, yeah. And there's a lot of monitoring platforms coming out all the time, right? I mean, it's like, everybody's got sort of some new flashier, similar way. I mean, fundamentally same problem, though. Yeah. So we got a question. I like this question a lot. Is it possible to evaluate, say, correctness or groundedness before issuing an answer to avoid unwanted hallucinations? When you ask the question possible, when it comes to LLMs, the answer is always probably or yes, in this case. I mean, you know, one of the metrics we looked at in Langsmith was latency, right? So the idea is that these requests take some amount of time, and the evaluation is going to take more time than the initial request, at least as much time as the initial request, likely more. And so the idea is, yes, you can do that. And perhaps even you should do that, but there is going to be a latency hit that you have to do that. There are platforms or tools like guardrails or like guidance that can help sit in that loop better. There are also API services that can do this. But yes, you could do like a harmfulness pass. What we'd really want to do in terms of correctness, groundedness, is perhaps build a classifier that was lightweight and robust that would sit in that loop for us in order to have the least amount of latency. There's also kind of an interesting push-pull where we want to stream responses, but if we're streaming a response, we don't have the full response to necessarily check for the complete harmfulness or correctness. There's a lot of moving parts here, but yes, it is possible, perhaps not completely feasible right now through these kinds of tools, but there are others that are looking to accomplish that exact task. Yeah. And besides, what do you even tell the user as you're as you're doing this or even as you say oh no i don't want to i don't want to output that you just have the i am thinking not you know beeping thing up there you know something like that i need a little time to think uh that would be hilarious so uh reuben reuben asks is this replacing the RAGUS library? What's the deal? Yeah. Kind of though. So RAGUS is a very good tool, but it's kind of like a collection of good ideas. Those ideas are not unique or new. And so Langsmith is going to help you use RAGUS-like evaluation. It does not, though, completely replace RAGUS if you mean specifically RAGUS metrics, the way that RAGUS has implemented them, right? It's going to be able to do the same things and achieve the same goals using their own implementation, but it is not literally going to call the RAGUS library tools. And so if you have an affinity for those or they work a particular way that you enjoy, this is not gonna replace RAGIS. It also works well with tools like RAGIS, right? We can define our own evaluation metrics, our own custom evaluation metrics. And we could just, you know, we could just hide, tuck away a little RAGIS assessment in there, um, in order to get those scores as well. So there's a number of ways that we can integrate and use them together. But for the most part, I would think of this as a replacement generally for ragas, plus a bunch of other stuff, even if it's not specifically replacing like every single API call for. Yes. Yeah. And like when I was looking at the eval metrics they're not exactly the way they're calculated and things they're not the same you know even if they're trying to measure the same thing so it's it's sort of like you know put as much eval on there as makes sense to try to understand the problem and uh but yeah as a baseline just measure something chris i love this question how much did this presentation cost i think about like uh 85 cents to a dollar okay okay that's pretty solid yeah and that and that to be clear that's with uh you know repeated reruns and that's with kind of running through the notebook a few times to make sure that it would work for you guys well, but it was still coming under a dollar. So pretty, pretty reasonable. Yeah, with that sweet, sweet eval. Yeah, with the eval. And I ran the eval probably about six or seven different times. Yeah, yeah. Okay, so last question that we have time for today. Can we set up thresholds and alerts, like for latency, for instance, to trigger Ops team actions? Okay. So the answer is you can build a system that will do this. As of right now, I do not believe that Langchain supports this directly out of the box, but you can through the Langs client API, you can build a system that would do this. So that would look for what's my P50 latency in the last three hours, and if it exceeds some threshold, run a job, right? So you can absolutely build a system that does this because they expose all these metrics through the API. I don't think, or sorry, through the client SDK, I don't think that there's any specific automated triggering that's built into Langsmith. At least I haven't found it or read about it in their docs. But you can absolutely build this yourself pretty straightforwardly. You just have to retrieve the metrics that you care about in the timeframe you care about. And then we're if else and from there. Nice. Nice. And that addresses Pano's API question as well. We're going to go ahead and collect any remaining questions and try to get some posts out next week on LinkedIn to answer them. So thanks everybody for your participation today. Thanks Chris, for joining us for Q&A, showing us how LangSmith is done. All right, everybody, that is pretty much a wrap for today. Thanks for joining us. This event has been brought to you by AI Makerspace. And of course, if you want to go deeper on LangChain, on LangSmith, we've got an upcoming cohort in 2023 for LLM Operations. This is going to be our fourth cohort, and we're really dialing in the way that this thing comes together in just four weeks to give you the broad picture of everything you need to build, ship, and share production LLM applications. From Langchain to Llama Index to WanB to Langsmith, and even more, we'd love for you to put your application in and potentially join us. If that's not a great fit for you, or you need a little bit more ramp up, you may want to consider our LLM engineering course that dives into the details of what happens inside an LLM, how it's trained, how we fine tune. That class is contained completely within the Google Colab notebook and doesn't require putting together a lot of different Lego blocks. That one launches January 9th. And if you're interested, that'll be cohort two and it's been going super well so far. If you have any questions, feel free to reach out to us anytime. Greg at AI Makerspace, Chris at AI Makerspace.io. And until next time, keep building, shipping and sharing, and we'll do the same. Thank you so much, everybody. See you soon.
LangSmith: Operating Production RAG Applications
GPT-4 Summary: Step into the world of Generative AI and master the art of evolving LLM applications from prototype to production with our dynamic event! Discover how to effectively baseline and evaluate your system's performance, manage key metrics like cost, latency, and token count, and implement continuous improvement strategies. Delve into the tactical application of these concepts using the LangSmith platform, exploring its comprehensive monitoring, evaluation, annotation, feedback, testing, and debugging features, all built on the robust LangChain framework. Whether you're looking to optimize your RAG systems, fine-tune LLMs, or ensure your application's scalability, security, and reliability, this event will provide you with the insights and techniques to enhance efficiency and performance in the fast-paced world of Generative AI. Join us to learn how to navigate the complexities of operating and improving your LLM applications in production!

Event page: https://lu.ma/Langsmithllmapps

Have a question for a speaker? Drop them here: 
https://app.sli.do/event/pchsPem1YK15a5CMfv6DYN

Speakers: 
Dr. Greg Loughnane, Founder & CEO AI Makerspace.
https://www.linkedin.com/in/greglough...

Chris Alexiuk, CTO AI Makerspace.
https://www.linkedin.com/in/csalexiuk/

Join our community to start building, shipping, and sharing with us today!
https://discord.gg/RzhvYvAwzA

Apply for the LLM Ops Cohort on Maven today! 
https://maven.com/aimakerspace/llmops

How'd we do? Share your feedback and suggestions for future events.
https://forms.gle/N4L7KMmH2Qf2sMFn7