 Hey this is Lance from Langchain. This is our third video focused on Langsmith evaluations. So the first video kind of laid out why evals matter, why they're interesting. The second video laid out the core Langsmith primitives that we're working with. So now let's actually jump into some code. So again this is just the overview of the kind of eval landscape that we've talked about previously. There's data sets, there's evalu about previously there's data sets there's evaluators there's tasks care about and there's you know how do you apply your emails so all i've done is if you go to smith online.com this will be an opportunity to sign up if you haven't done all done so already i've already signed up of course so now this is showing my my workspace which we're going to talk about later um i've done some tip installs pip installed installed langsmith open ai olama no line chain install here we're just going to work with langsmith directly we're not going to involve langchen at all um so here i'm setting the api key that i got when i signed up and i'm also setting this other environment variable to enable tracing and i'm going to find a new project called test so this light chain project basically sets up a new project that I'm going to work in, and you'll see kind of how that's interesting very shortly. So here's like a first question you might ask, how do I build my own data set, right? It's a very simple, reasonable question to ask. Now let's say we want to build a data set of question answer pairs for this new blog post on the Databricks model DBRX. Really cool release, state-of-the-art open source LLM, a lot of nice detail in this blog post let's say i want to build a question answer data set based on this blog to test the system i have for answering questions in general right this is a very popular use case so what we're doing here is i'm kind of graying out everything we're not focusing we're only focusing on manually curated data set that's it so what i'm going to do i've already kind of gone through the post and i've curated a few questions and a few answers to those questions and this is just a good old pandas data frame that's it now what i'm doing here is from like smith important the client and i'm going to find a new data set dbrx so this is the data set I want to work with and what I'm just I'm just calling this create data set giving it a name giving a description I'm passing in the inputs and the outputs that I specified up here that's it so I'm running this and that runs now here's where I can hop over to Langsmith and let me move this if I go over to so you can kind of see a few different categories projects annotation queue deployments we'll talk about all that later don't worry about that for now go to data system testing and you can see a whole set of I have a ton of data set that I've been doing a lot of work but let's try dbrx that's the data set we just created I think so here it is so okay we can see created modified now let's just click on one of these we can actually just see here's the input question here's the answer and you know so that's kind of nice we can look at our data set here this test thing tells us have we done any evaluations on it. We've not. Just a set of examples. That's really it. So it's pretty nice. Now, let's say I want to update this. So I want to add a question. Again, just call create examples, data set name, or ID. There it is. I go back. It shows up. Easy. Now, I also want to say, okay, what are the different data set versions? I can rewind. Okay, okay, what are the different data set versions? I can rewind. Okay, that's what it was originally. This is what it is currently after my update. Let's say I want to edit it. I can actually go to an example. I can edit it here. There we go. Easy enough. Cancel out. That's really it. So we can go back here what you can see is we define a set of question answer pairs and we've used a linesmith sdk to just create a data set from them directly we've shown that edited the data set and we've shown that it has versioning that's kind of it, if we were to go back, and if I click on create new data set, I've saved that eval CSV. Let's say I want to create a new one. I test data set. I'll call this a key value data set. We talked about that previously. The inputs and outputs are just key value pairs, question, answer. You can see it's kind of all automatically populates boom and there it is I have it from a CSV as well that's really it it's super simple I define my inputs my outputs I've used a client to create a data set I've edited it I've shown how to look at the versions I've shown how to create a data set from a CSV using the UI that's really it again that's the foundational piece of building developer curated or manually curated data sense and we're going to be talking about kind of how to build on this next
Manually Curated Datasets | LangSmith Evaluations - Part 3
With the rapid pace of AI, developers are often faced with a paradox of choice: how to choose the right prompt, how to trade-off LLM quality vs cost? Evaluations can accelerate development with  structured process for making these decisions. But, we've heard that it is challenging to get started. So, we are launching a series of short videos focused on explaining how to perform evaluations using LangSmith.

This video introduces how to create, edit, and version your own evaluation dataset using the LangSmith SDK.

Documentation:
https://docs.smith.langchain.com/evaluation/faq/datasets-client

Notebook:
https://github.com/langchain-ai/langsmith-cookbook/blob/main/introduction/langsmith_introduction.ipynb