 Hi, Harrison from Langchain here, and today I want to talk about a series of features that we're releasing as part of Langsmith. So if you haven't used Langsmith already, it's our LLM systems ops platform for logging, monitoring, debugging, testing, evaluation of your LLM apps. And we're releasing a series of new features specifically around production monitoring and automations. So, Linksmith is really handy for getting a sense of what's going on in your application, and that doesn't just mean what's going on during development time or in offline mode. It also means what's going on when your application is live and in production. And so we've added a bunch of new features to help you understand what's going on and then take action going to have detailed videos on all these features, and then also some more end-to-end use case videos on how to combine and use these features for very real situations. So as an overview of what we'll be covering, first we're going to talk about filtering. So as you get more and more logs in your production service, you need to be able to filter into subsets that you want to explore really easily, and we've added a really good filtering experience. We're then going to talk about some of the monitoring charts that we've added and all the things that you can do with them. So this provides a really great way to get an aggregate view of what is going on in your application. After that, we're going to talk about threads. So one of the main UXs for LLM applications is chat. And so being able to group different turns of the conversation together into a sensible view and have a kind of like a great bird's eye view of that thread or that conversation is really important. And so we've added a view specific, not at the trace level, but at the thread level. And finally, we'll dive into automations. And so automations are basically ways of taking filtered subsets of data automatically and doing things with it. So one thing that you can do with it is you can send it to a dataset. Another thing you can do with it is you can send it to an data set. Another thing you can do with it is you can send it to an annotation queue, and I'll explain what an annotation queue is. And then the third thing you can do with this, and this is a really big new feature we've added, is online evaluation. So with online evaluation, you can define a prompt to run over some subset of traces and leave some feedback and automatically evaluate your runs that you see coming in to the production traffic. So I'm going to cover all these things and then I'll also cover a few use cases and so what you can do with these and and really like real world problems that you can solve with this concept of production monitoring and automations. with this concept of production monitoring and automations. All of these features are part of LangSmith, which is our SaaS platform for doing logging, monitoring, testing, and evaluation. You can use LangSmith whether you are using LangChain or not, so it is completely independent. In order to sign up for an account you can go to smith.langchain.com and you can sign up for an account for free. You will probably want to do that before we continue with the rest of this video. Once you've done that, come back and jump into a future guide. Let's get started!
Introduction: Monitoring and Automations Essentials with LangSmith
You’ve shipped your AI application to production and users are starting to interact with it, congrats! But as with all software products, the hard work *starts* on launch day. How are people interacting with your chatbot? Is your application performing well or hitting rough patches? Who was affected, and how bad was it? Once you diagnose the problem, how are you going to fix it?

In this video series, we’ll show you how to use LangSmith to:

1. Monitor your application with aggregates and drill downs, giving you the trendline and fine-grained filtering to understand your application performance.

2. Set up automations to take action on production data, either for online evaluation, data set construction, or human annotation.

3. Enhance performance of your application by leveraging the production data and feedback collected from monitoring and automations. We’ll run through an example of how you can incorporate this feedback to optimize your application.

Documentation: https://docs.smith.langchain.com/monitoring
Blog post: https://blog.langchain.dev/langsmith-production-logging-automations/