 One of the actions that we saw that we could set up automations to take is to send runs to an annotation queue. But what exactly is an annotation queue? That's what we're going to cover in this video. So annotation queues are a user-friendly way to quickly cycle through and annotate data. And so let's see exactly what that means. So if I go to Langsmith and I go to this little sidebar on the left, I can see that I have on the second item it's this annotation queues tab. So if I go to Langsmith and I go to this little sidebar on the left, I can see that I have on the second item, it's this annotation queues tab. So I can click on that and I can see a list of all the annotation queues that I've created. If I click into one, I'm then presented with a kind of like user-friendly way to annotate these runs. So let's break down what I have going on here. Here I have the inputs and the outputs of the run. And so this right here is actually the annotation queue we have set up for the magic filter thing. So if you remember, if you go all the way back to the filtering video, we have a way to automatically kind of like create filters from natural language. And that's what's going on here. So here I can see that I have inputs, I have outputs, and then there's a bunch of stuff I can do. So let me break that down. The first thing I can do is view the run. So if I click view run, I open up a sidebar and I get this very familiar looking debugger tracer mode where I can see everything that's going on. I can see what project it came from, and if I want to, I can navigate back to that project. Up in the top, I have a sense of how many runs I have left to annotate. And so we'll walk through going through the annotation queue, but this provides an overview of how many runs are actually present. Over here is where I can leave feedback. So I can leave feedback tags by clicking on a tag here. I can add a new one or choose from any of the ones that I've set up. I can also add a note. So here it looks like the user input was actually already a query, so maybe I'll say something like already a query and leave a note'll say something like already a query. And leave a note just for anyone in the future to kind of like come and look at. And all of these are saved with this run. So if I query this run programmatically or anything like that, they're all attached to it. So that's leaving feedback. And now let's talk about cycling through these data points. So there's a few actions that I can take for this data point. First up, I can add it to a data set. So if I click add to data set, I can then choose the data set that I want to add it to. I can also edit it before I do that. So these blocks here are editable. So if I do something like runs with name text to playbook or something like that. I can then click Add to Dataset, and it will add the edited version. So this is really useful when you send, and I'll talk about automations and some of the automations you can do, but you can send runs with negative feedback here and then you can correct the output to something that it should be and then you can send that corrected output to a data set other things that I can do I can move this data point to the end and so that just moves it to the end of the queue so it will move it back here so if I click this you'll notice that the number of things in the queue doesn't change but I go to the next data point. Alternatively I could click done and this will mark this run as done and you'll see that this number will actually go down. I can also cycle through the runs here so here it looks like basically the same thing was sent a bunch this was probably because we were testing something out. But here I can get to some real runs. And so I can see that I can navigate with these buttons forward and backward. And so if I want to just jump through and see what's going on, this is a user-friendly way to do that. So that's an overview of annotation queues. And remember, these can be used in automations. So if I go back to my project and I set up an automation, I can click here in action, send to annotation queue, and I can select an annotation queue to send this subset of runs to, or I can create a new queue. That's it for annotation queues. Thanks for watching.
Annotation Queues: Efficiently manage data review with feedback tools
If you’re trying to improve user experience, you may want to drill into traces with negative feedback or gather traces with positive feedback to use as few shot examples. With LangSmith, you can define filtering parameters and automatically send traces to an Annotation Queue. Annotation Queues allow you to closely inspect traces and tag them with different criteria, helping you categorize and layer on human feedback.

Documentation: https://docs.smith.langchain.com/monitoring/faq/annotation_queue
Blog post: https://blog.langchain.dev/langsmith-production-logging-automations/