 In this video, I'm going to talk about automations. So, so far we've covered a bunch of functionality for how to allow you to dive in manually and pinpoint data points and view data points and debug things and all that is great and looking at your data is so important and you absolutely should do that. But oftentimes you maybe want to set up some automations to do things over the vast number of runs that you're getting in production automatically. And that's exactly what we've built. And so this is going to rely pretty heavily on filtering. So if you haven't watched the filtering section, I would highly recommend watching that first. Because basically the way that automations work, and here I'm going to switch over to Linksmith, but the way that automations work are that you can go in to a project, you can apply some filter, and then you can set up a rule to run over a random subset of those data points. So let's break that down. So here I'm in the chat link chain project. This looks over all runs in our Chatling chain documentation bot. I can create a filter. So again, this is where the filtering section comes in handy. But let me create a really simple one that is something like all top level runs where feedback, where user score is positive. And so these are all runs that got a thumbs up from the users. And so I can look at these manually. I can inspect them, that's great. But I can also add a rule. So I'm going to click on this, and it's going to pop up a little sidebar. I can give a name to this rule. So let me give thumbs up as a name. I can also adjust the sampling rate. So by default it's set to 1, but I can drag it to be anything between 0 and 1. This is going to represent the percentage of the runs that meet this filter that I'm going to apply the action to. So let's talk about actions. So there's three actions that we have. Send to annotation queue, send to data set, and then online evaluation. I'm going to deep dive into all three of these later on, but at a high level, sending to annotation queue allows you to send data points to an annotation queue where humans can go in and review them in a really easy way. Sending to a dataset allows you to send runs to a dataset, and you can use this dataset for testing and evaluation, you can use it for few shot examples in production, you can use this data set for testing and evaluation. You can use it for few-shot examples in production. You can use it to fine-tune models. It's basically creating a collection of data points. And then online evaluation is going to be a very deep dive section later on, and what that is is it's going to run an LLM over this subset of runs and then apply some feedback automatically to that based on the LLM scores. So this is how you can set it up from the project. You can also view and edit all these rules in your settings page. So if you go to settings and then if you go to rules, I can see that I've created a bunch of these rules already. I can edit them and delete them from here as well. So I can go in and edit. I can edit them and delete them from here as well. So I can go in and edit, I can see what we've created, I can edit it if I want to, or I can just delete the rule. And so you can manage all these automations from this page. That's an overview of this automations feature. The real power comes in with with what exactly you're doing with the data points. And so I'm going to talk about the annotation queue and online evaluation in deep dives in the next two videos, because I think those are really important to understand. And then I'm going to talk about a few use cases that you can do. So end to end kind of like framing of problems that you can do with these automations. Let's go to the next one.
Automations: Streamlined data workflow for Datasets, Annotations, and Online Evaluations
Production AI applications generate a large volume of data, and in order to extract insights and improve your app, you’ll need a way to sift through information quickly. While it’s useful to look at data by hand, creating automations to route your data can save a considerable amount of time.

Documentation: https://docs.smith.langchain.com/monitoring/faq/automations
Blog post: https://blog.langchain.dev/langsmith-production-logging-automations/