 Hi, this is Lance from Langchain. We're continuing our Langsmith evaluation series focused on dataset splits. So let me start by giving kind of a motivation for why we might want to use dataset splits. I have a RAG app that I've been testing throughout this series focused on the Langchain expression language documentation. Now very recently updated those docs from v0.1 to v0.2. So what's going to happen now is I have an eval set of 20 questions related to v0.1 docs that I now want to store as a split. And I want to see if I create an app using these newer v0.2 docs, are they backwards compatible in terms of evaluation with my existing eval set? Or do I need to change my app to load information maybe from other sources because basically my worry is when we upgrade the docs things may have shifted around so what I was loading from here uh may be insufficient now with these vr.2 Docs so basically my concern is that the data loading is a bit different and I may have regressed in my existing eval set with this dataset upgrade. So what I'm gonna do is I'm gonna go ahead and in my datasets here, I'm gonna create a new dataset. I'm gonna select a CSV. I'm gonna pull this in. So this is my existing dataset. So this is LCEL, question and answer, and I'm going to call this splits. Splits for L-C-E-L QA, RAG, QA. I'm going to call it that. It's a key value data set. You can see my input and output fields, and that's great. So I create this data set. Nice. That's all done. Now, here's where things get kind of interesting. dataset. Nice. That's all done. Now, here's where things get kind of interesting. I'll move this over here. So I have 20 questions here. I'll open this up so we can look. 20 questions here are related to, are from my original dataset. So that's all these guys. So this is my initial eval set that I've been using historically throughout this throughout the series related to to rag and language expression language i'm going to add these to a split and i'm going to create a new split and i'm going to call it um i'm going to call it uh lcell v 0.1 to to identify that these uh question answer examples were taken from the originals v 0.1 docs um cool so now i have this split that's been created right here now i can go back and this is my original so this is the original set of 25 examples now i have five new examples which you can see down here. Cool. So that's these. These are examples that I've added. I'm going to call this LCEL V0.2. So these are five examples I just very quickly put together that are on that main page of the v0.2 docs. So it's kind of like a test. These should definitely work reasonably well because I've created them newly. My bigger question is, do I see regression on the initial split? So right now what I have here is two data splits. So one, these are five questions I put together pretty quickly based upon the newer structure of the docs. It's kind of like a test. So this is actually what the new structure of the docs looks like. This is the new landing page for language and expression language. And you know, it's pretty nice. It has like lots of rich information here. That's great. Now here is the old landing page and it's just different, right? Different in structure. This old landing page kind of fanned out to a bunch of subsections, which is where my initial eval set drew all its information, right? So if I kind of zoom back out, what do we have here? We have two splits for my dataset. This LCELV01 is my initial set of questions, my initial 20 questions. And this LCLV2 are five new questions that I derived from this newer version of the docs. So this is kind of like my earlier original eval set. This is a new eval set. But they're all related to language and expression language. I'm just going to store them as two splits within this project. Cool. So I've created a RAG app here. That's all set. I have my RAGBOT. Now, we've used this previously. That's all set. Let me kind of skip through some of this. And the key point is simply this. Right now, all I need to do is I can just take this dataset name here. Cool. And I can kick off an evaluation. I can specify the two splits I want to work. Great. So both evaluations ran. I ran one on the newer split. I ran the other on the older set of questions, my original set of eval questions. So cool. Here's the two results. I can see the split names here. So this is V2 or the newer ones, V1 or the older ones. Performance is a lot worse on the older ones. It looks fine on the new ones. So what this tells me is I can investigate this a little bit more deeply, and I can see that performance is pretty bad on my original set of eval question-answer pairs. And what this kind of would tell me is, okay, I need to do a little bit of work to make sure that my document loading within the new doc structure, as you can see what we do right here, see what we do right here, we should then make sure that this is actually gathering all the information necessary to actually perform well on our individual eval set. So if you actually go to the docs here, we can see the doc structure has changed a lot. So it's very possible that we're no longer kind of getting the right documentation or pages necessary to answer all the questions that are relevant. So this kind of thing will be really important if, for example, I have a production app that's loading from set of documentation, the documentation structure changes. I want to actually determine how much regression do I see with this documentation change? And how do I then need to modify my data loading to actually make sure that I have full coverage on the questions in my original eval set? So that's kind of a good example of where you might use splits. Also things like partitioning, you know, test examples from training examples, if you're doing fine tuning, is another very kind of very classic application for using splits. But anyway, this really just shows how you can actually set them up using the UI really easily. Thanks.
Dataset Splits | LangSmith Evaluation - Part 22
With the rapid pace of AI, developers are often faced with a paradox of choice: how to choose the right prompt, how to trade-off LLM quality vs cost? Evaluations can accelerate development with  structured process for making these decisions. But, we've heard that it is challenging to get started. So, we are launching a series of short videos focused on explaining how to perform evaluations using LangSmith.

This video focuses on dataset splits, which allow you to partition datasets and evaluate on specific splits. 

Docs on how to create/manage dataset splits: https://docs.smith.langchain.com/how_to_guides/datasets/manage_datasets_in_application#create-and-manage-dataset-splits

Docs on how to evaluate on a dataset split: https://docs.smith.langchain.com/how_to_guides/evaluation/evaluate_llm_application#evaluate-on-a-dataset-split 

Try it out in LangSmith: https://smith.langchain.com/