 Hi, this is Lance from Langchain. We've heard a lot of interest from users on evaluation in recent weeks and months, and so we want to kick off a short series laying out how to think about evaluation from scratch and how to implement it yourself using Langsma. So to kind of set the stage, when new models are released like Cloud3, you often see a bunch of public evaluations reported. So on the left here is the Cloud3 blog post showing a bunch of different evals in the various rows and compared to other popular LLMs in the various columns. You've also seen maybe things like Chatbot Arena, which now has Cloud3 Opus at the top. But the question here is like, what are these evaluations? How to think about them and how could I implement them myself? So maybe a nice way to think about evaluation is just four pieces. There's a data set, there's some kind of evaluator, there's a task, and there's some means of interpreting the results. So let's actually make this concrete and look at the various evaluations that have been run on some of these public models. So human eval is a really good one. It was produced by OpenAI back in, I think, 2021. It has 165 programming problems. So it's basically related to the task of Cogen. You can see that at the bottom. And what's interesting is the evaluation method here, you can think of it in two ways. What's like the judge? So who's actually judging the result? And like, what's the mode of evaluation? In this case, the mode is like ground truth. There's a ground truth correct answer for every coding problem, and you're using unit tests, some kind of programmatic way of specifying correctness. Interpretation typically just reported as bar charts. In this case, I'm showing some results from the recent Databricks model, which they report a human eval on. But let's look at another one. So here's an interesting kind of comparative example on chatbot arena so in this case there's actually no static data set this is more dynamically generated from user interactions and the way it works is a user's presented with two different llms they prompt them both and they choose which response they like better so it's more of like an arena or like a a battle format in that sense. So again, in this case, the judge is a human. The mode, in this case, is not really ground truth so much as a comparative assessment. In terms of metrics, they often report these pairwise plots, which basically show one model versus all other models. And then the statistics tell you the likelihood of one model beating the other. They roll these up into things like elo scores which kind of tell you the likelihood of a model beating another one um kind of taken from chess so anyway you can see that you can look at different evaluation like benchmarks using these four different kind of buckets and just group them that way and think through them in that way. But we've kind of seen an interest in personalized testing and evaluation. So for example, like, of course, models are, you know, published with, you know, hundreds of different public benchmarks, but people often want to build their own benchmarks and kind of hack on and test models themselves. We've actually seen some interest in the community around this. So Karpathy tweeted about one nice benchmark from a scientist at Google DeepMind. Will DePue from OpenAI mentioned there's kind of a lot of opportunity in better evaluations. So, you know, if you kind of talk about those four buckets and break them down a little bit, there's a few different things to kind of cover here. There's a lot of surface area for building your own evaluations. So when you think about data sets, there's a few different things to kind of cover here. There's a lot of surface area for building your own evaluations. So when you think about data sets, there's a few categories. Like one is manually curated, like we saw with human eval. Build a data set of question answer pairs or like code solution pairs, right? So there's like highly curated, you define it yourself. Another is like if you have an app out in production, you have a bunch of user interactions with your app you can roll those into a data set for example of user logs and you can use lms to synthetically generate data sets for you so these are all really interesting modes of data set kind of creation now in terms of evaluation we saw examples of human as a judge like in the case of chatbot arena in that case with comparison as the mode we talked about using like unit test or heuristics as the judge against like a ground truth correct code solution in the case of human eval you can also use llns as judges and there's a lot of cool work on this um llns as judges can you know judge for general criteria which you might think of as reference free like there's no ground truth but you give the lln like I want to assess a few different things like, you know, brevity or something. So it's kind of like a reference-free mode of evaluation. And of course, an LLM can also judge or an answer relative to ground truth. So the final piece here is thinking about like, how are these typically applied? You can think about a few different categories. Unit tests, evaluations, and A-B testing. So unit testing are kind of like simple assertions about functionality. These are very routine in software engineering. They can be run online to give an application feedback. It can be run offline as part of, for example, CI or other kinds of evaluation. You can also have, like, again, like we talked talked about before a proper evaluation with a judge in this case it's not just like um you know maybe a simple assertion in this case maybe it's a more involved like human feedback or ln judge and again we talked a little bit about human evaluation and also ln based evaluation and then ab testing this is just comparative evaluations popular one here's regression testing, looking over time, or experimental testing, assessing different parameters. So a question you might ask is like, well, how do I actually get started? How do I implement some of these ideas myself? And this is kind of where Langshan comes in. So the team at Langshan built Langsmith as a platform that makes it very easy to run evaluations. And so if you go to this link here, smith.langshan.com, you'll be prompted to sign up i've already signed up so you can see this is just my like linesmith page we're going to talk about this in a lot more detail in upcoming videos but the point is likes with makes it very easy to instrument various evaluations we have a ui and sdk for building data sets versioning them editing them an, and SDK for defining your own evaluators or implementing or using custom evaluators. And we also have the UI for inspections, for trace inspections, comparative analysis. And we're going to kind of walk through in a few different videos these ideas kind of very carefully so you understand all these pieces and how to do each one from scratch. And of course, very importantly, LangSmet does not require LangChain to use. But of course, you can use it with LangChain. So that's an important point of flexibility I want to highlight. And in the upcoming videos, we're going to be digging into each one of these bins really carefully and building up an understanding from scratch as to how to build your own evaluations. Thanks.
Why Evals Matter | LangSmith Evaluations - Part 1
With the rapid pace of AI, developers are often faced with a paradox of choice: how to choose the right prompt, how to trade-off LLM quality vs cost? Evaluations can accelerate development with  structured process for making these decisions. But, we've heard that it is challenging to get started. So, we are launching a series of short videos focused on explaining how to perform evaluations using LangSmith.

This video lays out 4 main considerations for evaluation: (1) dataset, (2) evaluator, (3) task, (4) how to apply evaluation to improve your product (e.g., unit tests, A/B tests, etc).

Getting started documentation:
https://docs.smith.langchain.com/evaluation