 Hi this is Lance from Lanxchain. This is the sixth video in our series focused on Lanxnet evaluations. So in the first video we just kind of laid out why values are important. The second video we laid out the core Lanxnet primitives. We then talked about how to build data sets. First developer curated. We built one for the Databricks model which is discussed in this blog post. So we built a question answer data set the database model which is discussed in this blog post so we build a question answer data set um and we showed how to edit that we showed how to append um examples to it we then also showed how to build data sets from user logs so if you have like an app in production you're you know getting feedback questions from a user you can basically plumb those into a data set um we then just talked through how to use an evaluator, in our particular case, a built-in Lancet evaluator to judge our app, which we built here. So we had a very simple app that answers questions about a given context. In this case, we plumbed in the blog post, and we're using GPT-3.5 to answer questions. We're comparing those GPT-3.5 generations or answers to our data set, which was manually curated. So that's really it. We used a built-in length of the evaluator called COTQA, which is very convenient for question answering. So that's kind of where we are. That's what we did. Now let's build on this a little bit. And this kind of gives you a summary of what we just did. Now let's say I want to do something different. I want to find my own custom evaluator. So let's say I want like a simple assertion that an answer actually was generated or not. Like was there some kind of like failure in generation of the answer? So is that to return an empty string? Or there can be other reasons why no one fails to produce a generation right so in this case i'm changing it up a little bit i stole my data set of question answer pairs derived from the databricks blog post but in this case i want my greatest be it's like a heuristic it's a hard-coded decision it's just like is there an answer or not right it's reference free there isn't like a ground truth there it's just like is there an answer or not right so it's more like criteria based uh grading in this case you can this is where it's a little bit squishy you think of this more as a unit test it's a very simple assertion for functionality um as opposed to it being an evaluation which is using like an lm as a judge or human as judge this is just like a quick test is there an answer present or not and of course this can be used you know an offline evaluation potentially something like ci right so it's like a nice test or sanity check that my system's actually working so how would i do that so it's actually really simple this little function is actually all you need but let me actually explain a bit about what's going on here remember our flow diagram let's actually go back up so there's this run and there's an example so the run is my app basically my app has some output right so the run has this dot outputs which is basically the the dict or the object that comes out of my app and And then likewise, my example, which I pass in evaluator, also has this dot outputs, which is basically everything that's available to the example. In my particular case, what I'm going to care about is, you know, in typical evaluations, I often care about just the answer. But the point is, when I'm doing evaluation, there's this example object and there's this run object. And these two things are being compared in some way so that's really it so remember i have an example and a run now if i build my custom evaluator i plumb in that example and run just like we just talked about and i can just easily fish out from the run which is my llm app i want to get the answer now let's actually go all the way back and make sure we understand what we just said so i'm going to go back to my app here's my little app answer database question open ai it returns a dict with this answer key that's where me that answer thing from so that key depends on my app in this case my app returns you can see it's a dict with answer in it that's it. That's it. So that's why in my custom evaluator, when I go down here, I just go to my outputs from the run. Again, it runs my app. The outputs contains everything output by the app. In that case, that output is a dict, but it has a key answer. So I'm fishing it out like that. And I do something really easy. If there's nothing there, then it's not answered. Otherwise, it is. As simple as it gets. Remember, I have this data set DBRX. I'm going to run on this. So again, here's my app. Basically, we already talked about this. Here's the data set. We already showed how to build that. This is my new little evaluator is answered. I'm going to create a new prefix for it. Add some new metadata, kick that off. That that should run pretty quick and I can go over to my data set it's already done so okay this is kind of interesting so now you see two evaluations the first one we showed previously using the built-in light chain evaluator now I have this new one you can see it has this new prefix and let's open it up and see so in this case those scores are all one because there is an answer every time. That's it. That's a custom evaluator. It shows up here in Langsmith. This scoring is arbitrary. We can choose different means of scoring. But in this particular case, I just chose one or zero. Shows up here. That's it. Pretty simple. And this is extremely flexible. I mean, this is just a function. It can be anything you want. Again, all you need is the ability to fish out your output. You could do string matching. You could do arbitrary different kinds of valuations. And you can also include, for example, LLMs in this to do arbitrary reasoning over your answers and produce scores. I've used this a whole lot it's extremely convenient very flexible and building custom values is extremely simple particularly once you understand that kind of that information flow that we showed here all that's happening is I have an example I have a run each of those has this dot outputs thing which gives you access to in the case of my run that output object my case was just a dict with answer and this dot output actually gives you access to everything in this example. That's really it. That's all you need to know. And that's really all I want to say about loaded customer value, where it's very powerful and very general. Thanks.
Custom Evaluators | LangSmith Evaluations - Part 6
With the rapid pace of AI, developers are often faced with a paradox of choice: how to choose the right prompt, how to trade-off LLM quality vs cost? Evaluations can accelerate development with  structured process for making these decisions. But, we've heard that it is challenging to get started. So, we are launching a series of short videos focused on explaining how to perform evaluations using LangSmith.

This video introduces how to define your own custom evaluation logic in LangSmith. 

Documentation:
https://docs.smith.langchain.com/evaluation/faq/custom-evaluators

Code:
https://github.com/langchain-ai/langsmith-cookbook/blob/main/introduction/langsmith_introduction.ipynb