 The last feature that I want to cover as part of this production monitoring and automation series is online evaluation. And this is a really cool and much requested feature that we've heard and so I'm really excited to dive into this. The basic idea of online evaluation is applying a prompt plus an LLM to assign feedback automatically to data points in production. And so I'm going to show how to do that. This is the documentation here, and I'll have a link for this in the description below. But to start, I'm going to jump back over to this familiar project that is ChatLangchain. And so these are all the runs we have coming in. And so one of the automations that we've set up is we want to tag all runs that have negative feedback with a vagueness tag. And so basically the reason for that is we want to look at all runs with negative feedback and basically determine whether it's gotten negative feedback because the question was vague or because the response was wrong. And so we could look at all data points by hand, but instead we're going to have an LLm do that and assign this tag and that's going to give us a good indication so the first part of that is setting up a filter and so this is covered in in the filtering video as well as the automations bit but basically i'm going to set up a filter for where feedback user score is zero so i set up this filter now i'm going to add this automation. I'm going to add something like vagueness as the name. The sampling rate I'm going to set to one. So I want to run this over all data points. And that's because Chatling Chain has a manageable amount of data points with negative feedback. And then I'm going to select this online evaluation component. And then I'm going to select this online evaluation component. So when I select this, I get this little button here called Create Evaluator. And so I then open up this tab here and I can see a few things. First, I can see the secrets and API keys associated with this online evaluator. So remember, this is using a language model. And so we need to specify API keys for this language model. So if I click in here I can see that I can specify my OpenAI key right here. I can then choose the model. So here it's using GPT 3.5 Turbo. I can change it to any of the provided ones if I want. I can also change the temperature should I choose. The main interesting part comes in when you specify the prompt. So when I go here, I can click set in line prompt and I can get this template that pops up. And there's a few things to note here. First is the template. So the template has two input variables and it doesn't need to use these, but it should use only these two, because basically what's happening is we're going to fetch the data from the run, and we're going to pass it into this prompt template, format that into a message, and then we're going to pass that to the language model. So here I can see that input and output are the two prompt variables, and those are exactly what I should be using. And these represent the inputs and outputs respectively of the run. And so if the input and outputs are nested, if they're a dictionary with multiple keys or something, that'll be rendered as a dictionary as well. So keep that in mind when you're designing this prompt. So here, if I want, I can go in and I can change the prompt. And then the other really important thing that I can do is I can actually attach a schema to this. And so this is important to understand. The schema has a series of arguments. Each argument will end up being a metadata key that we attach to the run when it's finished. So here I want to specify vagueness. So I'm going to change the name of this from correctness to vagueness. And this name, this name of this key, this is what will show up on my run as feedback. Then I'm also going to change the description. So like, is the user input vague or not? I can mark this as required or not. So if I want to like optionally kind of like let the LLM leave feedback, I can unclick this. And then I can choose the type. So boolean, string, numbers, nulls, integers, objects, arrays. I can choose all of that. And this will be the type of the value that's left as feedback. I can also add other things so if I want to do vagueness and correctness in the same thing I definitely could and these can be different types as well and then when that's all finished I can hit save and it will save this rule or I'll save this online evaluator as a rule that gets run over this part of, over these sampled data points. I can see that I've set this up here, and so I've already done this, and I can see that I get the vagueness tags coming in. And so if I filter in to a particular subset of runs, so here I can see, so this is a run. If I look at feedback, I can see that it had a user score of zero, and I can see that it also had this vagueness component of one. And this is actually left by the online evaluator. I've also set up another evaluator that randomly samples data points and tags them according to different categories. So this was labeled as conceptual. So hopefully this shows a few interesting ways that you can use these online evaluators to automatically tag, look at, classify, provide more insights into the various inputs and outputs that I have coming into my system.
Online Evaluation: Simplifying assessment of LLM responses
While it may be tough for a human to look at a large amount of data — it's quite easy for a language model! We’re excited to introduce Online Evaluation, which allows you to set a LLM and define a prompt to run over production traces. Online Evaluation enables you to more easily classify and spot qualitative trends or drift in your application performance.

You can evaluate traces based on criteria such as “insensitivity” if the LLM responded in an insensitive manner or give it a “topic” category to summarize the content of the response — the criteria is completely customizable.

Documentation: https://docs.smith.langchain.com/monitoring/faq/online_evaluation
Blog post: https://blog.langchain.dev/langsmith-production-logging-automations/