 In this video, I want to talk about monitoring. So it's great to look at traces individually, but oftentimes you want to look at aggregate statistics to get a better overall sense of what's happening inside your application. And we've added a monitoring tab to enable exactly this. So I'm going to walk through this in linksmith, but I'm also going to highlight this documentation that we have here, and I'll link to it in the description below. So in linksmith, you can go into a project that you have. I'm going to go into chat link chain, and these are runs that make up the chatbot over our documentation. I'm then going to click on this monitoring tab up here, and I'll see that I get these monitors of various statistics over time. And these are statistics that we find to be important when developing an LLM application. So here this is just a basic count of traces over time and of LLM calls over time. So here there's, you know, they correlate pretty heavily, but it's possible that I could these could not be kind of like as closely correlated. I can also see the trace success rates over time. So I see that I had a slight dip here. But other than that, it's pretty good. I can also see the trace success rates over time. So I see that I had a slight dip here, but other than that, it's pretty good. I can also see latency over time. I can see tokens per second. I can see feedback over time. And so I track user score and we also track vagueness as well. And so these are two different feedback measures that we can see over time. Tokens we track as well as cost. And so we've really emphasized this for LLM specific features that the above mentioned, as well as things like streaming. So how often streaming is used, time to first token, and things like that. So this is great for getting an overview of what's happening in your application. You can also zoom in and out. So I can zoom out to a 30 day time period and see this a little bit more over time. I can also zoom in more closely. And so here's what's happened in the last hour. Another thing that I can do is filter based on metadata or rather group based on metadata. I can do both. So when I click on metadata, if I click LLM, I can basically then have these results grouped over time. And so what's going on under the hood is that in ChatLangchain, we actually rotate between five different LLM providers. Anthropic, Cohere, Fireworks, Google, and OpenAI. And so I can track the various statistics within these groups. So this is really helpful for basically changing parameters of your application and then tracking it and seeing how it performs. So some interesting things that I can see are basically trace success rates. So here, when we had this dip before, I can actually see that it was Cohere that was erroring a little bit, and it caused some of the errors. I can also track latency over time. So I can see right here that OpenAI and Fireworks seem to be the quickest ones to respond. I can also track feedback. So here it's a bit more mixed. We don't get a lot of feedback on ChatLink Chain, unfortunately, but you can track feedback over time and you can basically see how different parameters are performing. And so this is really cool for tracking different aspects of your application. And here we show off tracking different LLMs, but it can be other features as well. You can have different prompts that you rotate through. And as long as you attach them as feedback, you can filter and group based on them and get these modeling charts. Another thing that you can do is filter into these subsets really easily. So here I can see that I had this dip in success, and so what I can do is this this corresponds to an increase in error. Right, so if I go here I can see that we have this spike here from Coherent to have a pretty high kind of like error rate. And so if I click on this data point what this is going to do is this is going to actually give me all individual traces that are in this data point on the screen. So here I can see that I have all the ones with failure. And so here if we check into chat, go here, I can see that I got a timeout error. And so it's really useful for diving in to different subsets of the data and really focusing in on what's going wrong there. That's pretty much it for monitoring. Hopefully, this shows you how you can use the monitoring dashboard to really track the performance of your application over time, as well as group different runs and track those different subsets over time, which can allow for some really interesting comparisons in an online setting. And I think that's important because oftentimes you can do a bunch of tests and we have a great kind of like testing framework as well but that's mostly offline and it often changes when you put things in production people use it in ways that you don't expect and so being able to track that performance in an online manner is super important and so these are some of the features that we've added to enable exactly that
Monitoring: Aggregate LLM stats and metadata grouping in LangSmith's interactive dashboard
Monitoring allows you to get an aggregate view of what is going on in your application over time. LangSmith provides monitoring charts that allow you to track key metrics — such as success rates, latency, feedback, cost, and qualitative characteristics with our new feature, ✨Online Evaluations✨

LangSmith also allows for tag and metadata grouping, which lets you mark different versions of your application with identifiers and view how they are performing side-by-side. This is helpful for A/B testing changes in prompt, model, or retrieval strategy.

Documentation: https://docs.smith.langchain.com/monitoring
Blog post: https://blog.langchain.dev/langsmith-production-logging-automations/