 So in this video, I'm going to walk through a more end-to-end use case of using LingSmith automations to optimize your application over time by constructing a few-shot example data sets. And so the example application that I'm going to build is an application that writes tweets for me. to build is an application that writes tweets for me. And so, you know, I like my tweets written in a particular style. And so I'm gonna build an application that can do that. And I'm going to show how to optimize that via a combination of user feedback and human annotation queue over time to build an application that writes more in the style that I like. So let's get set up. We can import, we'll use OpenAI for this. We can import Langsmith. And I'll create this really simple function. It's called tweeter. It takes in a topic, and it basically just calls OpenAI with this really kind of like just standard prompt. Write a tweet about topic, and then returns it. What I'm also going to do here is I'm going to pass in the run ID and we'll see why I'm doing this. But basically, this is so that I can specify the run ID ahead of time, which then lets me know how to add feedback for a run. So big part of this, I'm not going to actually create a front end for this. I'm just going to mock it out. But basically, the big part of this is collecting feedback from end users in some different environment um passing that into lanesmith and then using that to to kind of like optimize your application over time so here um i pass in the run id i pass an nba and i get back this tweet just watched an insane nba game those players are on another level hashtag nba hashtag basketball is life and then some emojis. I kind of like this one. Let's pretend for simplicity. I'm going to like tweets where it ends with an emoji. And that's a latent thing that's impacting whether I like the tweet or not. So let me leave kind of positive feedback on that. Let me try again. Let's do soccer. OK, so this doesn't end with an emoji. So even if this is a good tweet, I'm actually going to leave negative feedback on this. Okay, so that's the basic idea of what's going on. Now let's set up some rules in LingSmith that'll help optimize this over time. this over time. So what I'm going to do is I'm going to go into my project and I created a special project if you notice here I set the laying chain project to be optimization and so I'll go into optimization and I'm going to set up a rule that takes everything with positive feedback and adds it to a data set. So let me go into this filter I'm gonna add this filter feedback user score is one I'm gonna add a rule tweeting optimization sample rate of one I'm gonna send this to a data set data set name tweeting let me create a new data set tweeting optimization and let's create that let's save one about the NFL. And it doesn't end in an emoji, so I'm going to leave negative feedback or no feedback. At this point, I'll show it how to incorporate negative feedback later on, but for now, let's not leave any feedback. Let's do NDA again. Still no emoji. Let's keep on iterating until I get one that ends in a... All right, let's try to get one that ends in an emoji. All right, so this is proving a little bit more difficult than we go. So this ends with some emojis. So I'm going to leave now positive feedback on this. one that ends with an emoji. All right, this one ends with an emoji. Let's leave positive feedback on this. So now, what's going to happen is these things that I left positive feedback for, they'll start to get added to a dataset over time. And so these automations run every minute, and so I'm going to need to give it a little bit, but I should be able to go to a dataset and start to see these things pop up as examples in the dataset. So here we are in datasets and testing. I can search for tweet optimization and I can see that I have my two examples here. So if I click in I can see the input and the output and I can do the same for over here and I can see that they're the ones that end in emoji. So now what I want to do is I want to make my prompt a few shot example prompt and I want to start pulling these examples in and using them as examples for my application. Okay, so back in this notebook I am going to pull down examples from this dataset. So I'm going to use the Langsmith client, I'm going to list examples, set the dataset name equals to tweeting optimization, and again this is what I named the dataset. And I can run this and I can get my examples, which right now are two. And I can see that I have a bunch of information about this example. And the most important part is the inputs and the outputs. So I have here the inputs, topic soccer, and outputs, and then I have an output key and it's this tweet. So what I want to do is I want to use these as few-shot examples in a prompt. And so as part of that, this is what it could look like. So let's say, you know, we could take these examples and put them into some string like this. And so we'll have kind of like this input output pairing and then let's recreate our tweeting optimizer and here inside we'll actually pull down the examples so we'll refresh this each time this is maybe a little bit overkill because this will be another network call so you could do this outside but for this example I'm going to do it inside this function I'm going to create this string, I'm going to do it inside this function, I'm going to create this string, and I'm going to edit my prompt. So it's still going to say write a tweet about topic, but then I'm adding these new things. Here are some examples of how to do this well. And then I pass in this example string up here. And so hopefully, what we'll start to see as we give a few shot examples, it starts to implicitly kind of like learn what kinds of tweets I like and don't like. And so if we run it here and ask it to write a tweet about the Oscars, okay, awesome. So it added some emojis to the end. And so I can give positive feedback on that, and it starts to pick that up. One thing that I want to show now is how to do this same thing, but start incorporating kind of like negative feedback. And so there's actually a few like interesting ways that you could do this. You could create the same automation and basically send all rows with negative feedback to another data set and then include those in the prompt and be like, these are examples of tweets that the user did not like. So that's one thing you could do. But for a little bit more of variety, I'm going to actually send negative tweets to an annotation queue and then manually kind of like edit this. And so this shows kind of like the human in the loop component. So maybe, so let's see. So here, let me run this a few times until I get a tweet that I don't actually like let me change the topic to something like AI okay so here it doesn't end in an emoji so great I'm gonna leave negative feedback on that then what I'm gonna do is I'm going to go into my tweeting optimization project. I'm going to set up a filter for where my feedback user score is zero. And then I'm going to add a rule. Negative feedback. And I'm going to send this to an annotation queue I'm going to create a new queue which is tweeting optimization let me create that let me save that and awesome okay so now when I go to my annotation queue I should start to see these runs with negative feedback show up. So here we are in the annotation queues and I can click into the Tweeting Optimization annotation queue I just created. And here is the negative run that I got. And actually there's four of them because I gave some downloads before and those showed up. So here what I'm going to do is I'm going to edit them and then add them to a data set. So what I can do is I can just like edit this, delete that, now ends in emoji. Awesome. Now I can add it to a data set. Perfect. Done with that one. Go on to the next one. I'm going one gonna edit this one correct this to what I want it to be add to a data set let's add it to there we go keep on going all right this one doesn't have any so I'm going to remove uh the hash actually you know what I'm just going to skip this one. So I'm just going to be done with this without even adding it to a dataset. And here, boom, add this to a dataset. We're done. We're all caught up. Awesome. So now if I go back to the notebook, I can start to pull down the examples again. And I can see now I have a lot more examples. And if I run this updated tweeter anymore, let's choose a new topic like humans, it ends in emoji. If I go back to Langsmith I can go to my project, I can go to my optimization project, I can click into here, and yeah, this is the run that I had. So this is how we built an application that sets up links with automations that takes user feedback, takes ones with good user feedback, automatically adds it to a dataset. Takes one with bad user feedback, automatically adds it to an annotation queue. A human can then go in and correct it and it starts to build up this data set and this data set I'm then plugging back into the application and it's using that in future iterations to start tweeting in in this case tweeting but in your case it could be whatever application you're building it's optimizing that performance and making it better over time and so this is one use case that we're really, really excited about, and we built a lot of the functionality specifically to enable this.
Optimization Use Case: Build a style-adaptive app with LangSmith automations
All feedback is helpful, and when your users provide positive or negative feedback, you should leverage that information to improve future app interactions. Now with LangSmith, you can use automations to create valuable datasets from user feedback and integrate those data points as few-shot examples to improve your prompting strategy in your application. This use case walkthrough illustrates developing an application that learns to tweet in a style aligned with user preferences.

Documentation: https://docs.smith.langchain.com/monitoring/use_cases/optimization
Blog post: https://blog.langchain.dev/langsmith-production-logging-automations/