 Hi, this is Lance from LandChain. We're continuing our Langsmith Evaluation Series. We're going to be digging into online evaluation here a bit more. So if you recall, this is kind of a framework of how to think about evaluation in general. On the left here you see different types of data sets that you can work on. So you can work on kind of manually curated data sets, you can work on user logs, different types of evaluators. And what I really want to draw your attention to over here is the differentiation in terms of how you can apply evals. You can apply evals on production traffic. So that's like I have an app in production, and as my app is running, I'm doing evaluations of some sort on it. Or of course, I can also do evaluations on curated data sets or assertions, right? So online evaluations live in this top bucket and you can really see it here. So basically we're focused on production traffic. So I have an app in production and I'm evaluating something about that app while it's running. Now, a common thing that people like to evaluate is something like toxicity. So for example, does the user input contain any kind of personally identifiable information or toxic queries or something along those lines. So toxicity is a thing that oftentimes you want to be able to detect within an application as a developer. So I'll show an example of this. And I have a project that I've set called Rag Online Evaluation. And that's here, a few traces here you can see. And here's basically my Rag app. So with this app, I'm basically simulating user questions here. So here's basically my rag app so with this app I'm basically simulating user questions here so here's like an input question here's an input question and those are all logged to my project so that's all that's going on now let's say I want to create a new rule that'll evaluate whether the input question contains any toxic information or personally identifiable information so all I have to do is I go to Rules, and I basically say Add Rule. So this is going to allow me to create a new online evaluator. I just simply click here, Online Evaluation, Create Evaluator, and I'm going to use one of the suggested prompts that we have already. So right here you can see Tag LLM Generations. So this is a prompt that allows you to tag arbitrary inputs with a label or a score based upon some specified criteria. So you can see in the prompt template, it already is set up and I can choose my model. Let's say I want to use GPT-4 Turbo. It already has this kind of preset for me. So you're a classifier, they'll tag input text with a score based on supply criteria, and you're tagging based upon insults, threats, negative comments, any personally identifiable information, right? And you pass in the text. Now, what's pretty neat about this is I can actually test using this preview what this will look like on my particular chain. So with this preview, boom, I can actually see the text it's going to get is, for example, my most recent trace. So in my case, the most recent trace was this test case. How do I create a rag chain? And I also plumb in this OpenAI API key as like a test of an adversarial prompt or a prompt that contains personal identifiable information I'd want to flag. I have my score criteria down here, so I'll basically return a score, one or zero, and I'll also return the explanation for the score. So that's set. I just hit continue here, and then I'm gonna name my, I'll name this toxicity, or I'll call this guardrails. Input, name this toxicity, or I'll call this guardrails, input guardrails, and I'll save that. So now I have a new online evaluator called input guardrails that'll run on the inputs to my chain. So I go back to my app, and I've passed in this OpenAI API key, and I want to see whether or not my online evaluator can actually flag that. So I go back, and now my runs or my traces, you can see here's my app, here's the runs within my app, so this is kind of full trace we're looking at. We can see it's tagged with feedback. So the feedback I get is a score, in this case, of one. So if you look back at the prompt, one means that indeed there is a personal identifiable information within this prompt. So we can even look here, we can look at the explanation for this, open the evaluator run, and we can go down, and we can see, yes, it is scored as one. The test contains personal identifiable information as it includes potential API key, which is sensitive information that should not be shared publicly. So it's a good example of the ability to set up an online evaluator against a simple project. In this case, a mock rag app. It can run on the inputs of your app really easily, and it can establish any customizable guardrails or things that you want to flag for and return kind of a very simple customizable score. Yes, no, one, zero, in order to identify that the input contains information you want to flag. Thanks.
Online Evaluation (Guardrails) | LangSmith Evaluations - Part 21
With the rapid pace of AI, developers are often faced with a paradox of choice: how to choose the right prompt, how to trade-off LLM quality vs cost? Evaluations can accelerate development with  structured process for making these decisions. But, we've heard that it is challenging to get started. So, we are launching a series of short videos focused on explaining how to perform evaluations using LangSmith.

This video focuses on Online Evaluation focused on guardrails for PII or toxicity. We show you how to configure an LLM-as-judge evaluator with a customized prompt that can detect PII or toxicity on the inputs of an example RAG app.

Check out the docs for online evaluation: https://docs.smith.langchain.com/how_to_guides/monitoring/online_evaluations