 In this video, I want to walk through a few of the common workflows that we see around for how to use automations best. And so they're on this use cases page in the documentation here. And these are quick, high level overviews of those workflows. So let's jump into the first one, which is basically just sending bad data points into an annotation queue. And so the idea here is that you generally want to do this so that you as a human can look at things by eye. So the way that I would do that is I'd go into my project. I'll use the chat link chain project here. And I'd set up a filter to capture all runs that have bad feedback. And so I'd add a filter based on feedback. Here I know that user score is the one that I'm collecting from the user, and I'll filter to ones where it's zero. Once I've set up that filter, I'll then go into the rule. I'll say something like bad feedback. Sampling rate is generally good to have at B1, and this is because I want to look at all data points although if I don't want to look at all data points I could set it to 0.5 or something like that. I'll then click on annotation queue, maybe I'll create a new queue that's like bad chat lane chain runs and then I'll send things there. Then when things go into that annotation queue, I can look at them. Honestly, the main value to start is just like seeing what types of data points we're getting wrong. And then from there, I can leave more precise feedback. I can correct them and add them to a data set or something like that. The next use case I want to talk about is sending data points with positive feedback to a data set. So this is useful because we can assume that these data points are good, they're good representative examples, and that's great. We can use that as an example to test against to make sure that when we make prompt changes in the future we still do well. We can use that as an example to use in a few shot examples or fine tuning to improve the application. So here, I'll go in, I'll change, it's basically the same as before, but I'm just flipping the user score to one. And so now this is positive. I'll add a rule and this is like positive chat LC. Generally we'll take all of them. Again, you can change it if you want and then I'll send it to a data set. And so maybe I'll create a new data set and I'll have it be something like good chat LC examples. And then I'll select key value because I know that the inputs and outputs of these are usually dictionaries. And I will create it and save it. And boom, now I'm going to start building up this data set over time of examples with positive feedback that I can use in a variety of ways. The next one is sending child's runs whose trace got positive feedback to a dataset. Okay, so what does this mean? So this means if I go in here, let me remove this filter. If I go in here, let me click on one, I can see that there's a lot of runs here going on under the hood. And so what oftentimes happens is that I want to like this, this high level thing, it's great to have examples of that. But when it comes time to optimize what I really need, and by optimizing me, like use few shot examples or update the prompts, I really need few shot examples out at one of these levels. Cause like this high level input and output, it doesn't help me at this step where I'm kind of like generating an input and output. And so here we can see the condensed question. We can see that we get as input questions and then this chat history stuff, and then we get out this response. And so this is the inputs and outputs that I need in order to optimize this kind of like node of my chain or of my graph. And so this is really what I want to be building up examples for if my chain is more than one step. And if I want to be optimizing the individual nodes. Again, there's value in getting the end-to-end data set, but there's also a lot of value in getting data sets for the specific nodes. Again, there's value in getting the end-to-end data set, but there's also a lot of value in getting data sets for the specific nodes. So let's see how to do that. So here I'm going to remove this because I no longer want the top runs. I want the child runs. And so I want, I'm going to filter by name. I find that usually the easiest way to do it. And I'm going to filter by name. I find that usually the easiest way to do it. And I can see that it's condensed question. Filter there, I get all the condensed questions. And now what I want is I want the these, I want only the runs where the parent trace got positive feedback. And I'm basically assuming that if it got positive feedback at the end-to-end thing, then the step here is correct. And, you know, that's not 100% a good assumption to make, but I think it's a pretty solid assumption to make. Go to advanced filters. We can add a trace filter. And then we add the feedback. User score is 1 here. And so now I have this filter set up where I'm looking at condensed questions where it got positive feedback on the end-to-end thing. And just to make that even more clear, let me add, actually this automatically applies to the top level trace, so that's fine. Now from here, I can do the same thing as before. I can do like condense condense question data set I can send this to a data set and create a new data set. And boom! There we go. So now I'm going to start building up a data set of examples for this particular node in the graph. And so that's really powerful because, again, those are what I need to kind of optimize that call to a language model. It doesn't do me a ton of good to have a call end-to-end. That call at the node is what I want to optimize. The final one is just sending random data points to an annotation queue. And so this is good for just like making sure that you're seeing a variety of data points. So, you know, it's great to look at bad data points, but you want to keep on looking at, you know, you could just not be getting user feedback. That's very common. We don't get a lot of user feedback on chat link chain and so being able to just send random data points and then look at those random data points is pretty valuable and so what I can do is I'll just go back in I'll add some rule random data points I'm gonna set a lower sampling rate here because, you know, I don't want to look at all data points. I want to look at a random subset. I'll say like, yeah, sure, why not? Why not 5%? Send it to an annotation queue. I'll create a new queue for this. I'll go like random questions from chat link chain, and then I'll create that. And again, the idea here is, I'm just looking at a random subset. It's good for me to get an idea of what types of questions people are asking and stuff like that. So those are some of the high level common flows that we see people doing. I'm gonna dive into a few specific ones in the next videos.
Common Use Cases: Practical applications of LangSmith automation features
This video explores common use cases of the workflows covered in previous videos in the series, focusing on optimizing data quality through automated rules and human oversight.

Documentation: https://docs.smith.langchain.com/monitoring/use_cases
Blog post: https://blog.langchain.dev/langsmith-production-logging-automations/