 Hi this is Lance from LangSIM. This is the fourth video on LangSmith evaluations. So in video one we kind of laid out why evals are interesting and important. Video two we talked about the LangSmith primitives like the core foundational pieces to understand about LangSmith evaluation. And we just talked about dataset creation. We talked about an example of building a question answer pair dataset from this blog post on that recent Databricks LLM. So again, we showed how to build like a manually curated data set. Here's our inputs and our outputs. We use the SDK to build the data set and to update it. We also showed how to use UI. Now, a question I'd ask is, okay, that's great. But what if I have an app in production? question I'd ask is, okay, that's great, but what if I have an app in production? I have traces being, you know, logged over to Lanxmith. How to build a data set based on, for example, user logs? So, for example, let's say there's a bunch of generations or user inputs I thought were interesting. I want to build a data set from those so I can do future evaluations against it. So, let's do a kind of a toy example here. Let's create a new project. We'll call this DBRX. And what I'm going to do is I'm going to load that blog post. Here we go. So the same blog post we've been working with on the recent Databricks model. Now I'm going to define, here's my little toy app. So it's just answer a question. I'm using just OpenAI API. I'm not using Langchain at all here. Really simple. Answer the user question based on two or three sentences, given the context. That's really it. There we go. So let's say this is like an app. I have this in production somewhere. Users are just interacting with it. And I get a question. What are the main differences in training efficiency? And yeah, they have answers. Here's another one. Cool. and yeah they have answers here's another one cool so let's actually go over and we created a project for this move this over so go to my projects so you can see those two generations that i just ran here are going to be logged to this project dbrx so now we're in this project and we see those two generations you can look at the time it matches roughly the time that is right now pacific that's great so you know okay this is what we just saw it's actually kind of interesting to look at so here you can see this is the trace effectively so in this case the trace only has a single run so again we talked about there's runs and runs roll up into traces trace contains all the different runs in a given like application in this case it's a simple prompt response so there's only a single run and indeed the trace only has that single run in it so again you can kind of look over here here's your system prompt it plumbs in that whole blog post and you know the human prompt what here's the question and then the output so very simple now let's say these are examples of user interaction with your app you want to capture them for posterity or for evaluation or whatever just click on the two so we're in langsmith now um and you see this thing down here add data set so i click on that um i'm gonna move this over so'm going to call this a new dataset. Let's just call it like DBRX QA testing. Testing. There we go. Now it's automatically flagged as a chat dataset. Recall previously we built a key value data set based with basically question answer key value pairs for the inputs and outputs in this case this is derived from chat session so it's a bit a different data set type and this is useful for example if we want something like fine-tuning in the future the inputs are already in chat and serialized chat message format which is useful for for that which that's a more advanced use case we'll talk about later so you know that's really it we can just hit create and i'm going to go down to my data sets and let's just search dbrx boom and here it is you can see it's a chat data set it's derived from those inputs that my mock user provided um and you get the output so it's kind of basis set of input prompt output pairs from my app which i've now saved the data set which i could do work on accordingly and again just like before you can see you can edit these so for example here's a really good use case you have a bunch of user um you know interactions or inputs to your system but your system produced the wrong answers if you want to turn this into like an eval set you can actually edit these to be ground truth answer but you still have the real user inputs in there so it's just a very kind of common trick for building high quality evaluation sets um is to do this kind of logging and then potentially to clean the answers provided by the AI so set their then ground truth. So that's really it. So again, if I zoom all the way back out, what did we do? We just showed how to build data sets from user logs from this little toy app that I built that's answering questions about a blog post. So we're going to be kind of building on this in some future videos. But what we've really covered here is is ability to build data sets from logs and uh manually curate them from like question answer pairs or it can really doesn't have to be question answer pairs but now in this case i did um but you know developer curated versus in this particular case you know logs from a hypothetical user so two different ways of building data sets thanks
Datasets From Traces | LangSmith Evaluations - Part 4
With the rapid pace of AI, developers are often faced with a paradox of choice: how to choose the right prompt, how to trade-off LLM quality vs cost? Evaluations can accelerate development with  structured process for making these decisions. But, we've heard that it is challenging to get started. So, we are launching a series of short videos focused on explaining how to perform evaluations using LangSmith.

This video introduces how to create datasets directly from logs (e.g., user interactions with your app that are captured in LangSmith).

Documentation:
https://docs.smith.langchain.com/evaluation/faq/datasets-webapp

Notebook:
https://github.com/langchain-ai/langsmith-cookbook/blob/main/introduction/langsmith_introduction.ipynb