 Hey this is Lance from LanxChain. We're continuing our LanxMath evaluation series talking about repetitions. So the intuition here is actually pretty straightforward. We've talked a lot about different types of evaluations, for example that run on like larger eval sets that have different maybe complex elements judge evaluators. And in a lot of these cases we run an evaluation, we get some statistics or some metrics on our performance across the data set, We run an evaluation, we get some statistics or some metrics on our performance across the dataset. And you might ask the question, how reliable is this? You know, can I trust this result? If I run it again, can I reproduce it? And you can have noise introduced from different things. Your chain may produce kind of variable outputs depending on how you run it. Again, LLMs are largely, for the most part, non-deterministic. You know, your LM is judge evaluator. Again, it's using an LM, so there's some variability that can be introduced from the grading itself. So in any case, the idea of repetitions is a way to address this. Automatically run your evaluation n times to see whether or not it's consistent. It's very intuitive, it's very useful, and I've done this manually many times. But Langsmith is actually introducing a nice kind of new flag that's run simply with the SDK, where you can specify some number of iterations to run, and it's all nicely supported in the UI. So let's take an example case. This is an eval set I've already worked with related to language and expression language. This is an evaluator that I've actually used previously with a ragchain that operates on language and expression language documentation. And so this evaluator is basically going to grade an answer from a dragchain relative to a ground truth answer between 1 and 10. Okay, so that's all that's happening here. And this is my ragbot, which I'm just initializing with a few different parameters. I'm going to run it with GPT-4-0 with vector store, and I'm going to run it with GPT-4-0 with VectorStore, and I'm going to run it with GPT-4-Turbo without VectorStore. So these are just two example experiments I might run. And here's where it gets interesting. When I set up my evaluate function, just like we've done previously, I can just specify num repetitions and specify how many times I want to run this experiment. So in this particular case, my eval set has 20 questions. It's run three times. And so I run this, it actually runs 60 different evaluations. That's it. And again, I can run it on different configurations, just like I've done previously. So if I go over to the UI, here's my data set, just like we've seen before. Here's my experiments. And you're going to see something new and kind of interesting here. You're going to see this repetitions flag noted here. So what's cool about this, this allows you then, if you open up any of your experiments, right? So let's, for example, look at this experiment GPT-4 Turbo, you can see if you open up for any example, this is your input, right, here's your rag chain, you actually can see each repetition run. And so what's nice about this is that you can compare the answer for each of your repetitions. So that's kind of what you see here. And you can look at the grading. So you can see there's interesting differences depending on kind of the repetition in the answer itself, which can happen because certain LLM chains do have some variability, right? So the answers can differ by the chain. And also the grader given the same output can sometimes change, right right so what's nice about this is i can kind of go through my data set and i can actually investigate um cases of variability in the output um you know in this particular case you know one is grade one in one case repetition three is graded as one point eight point8, 0.7, right? So what's nice about this, these scores reported here are the mean of those three repetitions. So what's nice is these perform some smoothing across variability that's inherent potentially in your chain itself or in your LMS Judge Evaluator. It's a nice way to build better confidence in your results. In this particular case, this is working with a larger, more complex eval set. So in this case, it was a 20-question eval set. You can look at the examples here. These are kind of harder questions, so I do expect that the various experiments are going to have more trouble with them. And I'm using an LLMS judge evaluator. In this case, with custom criteria, we're going to be grading from 0 to 10. So again, that's like a more tricky grading scheme. There's more opportunity for variability there. And I can see in my experiments that indeed, if you kind of dig in, we showed some examples previously, but there is some variability across my grading. You know, grade of 1 here versus 0.5, 0.5. And, you know, the ability to run with repetitions gives me a little bit more confidence in the result. So it also makes it easier to compare with some confidence across different experiments when you've used repetitions to smooth out noise in your grader or in your chain itself. And really, that's kind of the intuition behind using repetitions. It's very intuitive. You can see I've run a number of these different experiments with three repetitions each, and this is kind of the aggregate of those means for each example being reported so i have a little more confidence in the difference in the difference between my various chains uh looking across these experiments relative to looking at a single trial or a single experiment that only ran on a single repetition so really that's just that's all there is to it's really simple and it's a very nice feature i've used it extensively just kind of manually but actually having is now as a feature and likes of it makes it much easier to run experiments with repetitions to build more confidence in your results thanks
Repetitions | LangSmith Evaluation - Part 23
With the rapid pace of AI, developers are often faced with a paradox of choice: how to choose the right prompt, how to trade-off LLM quality vs cost? Evaluations can accelerate development with  structured process for making these decisions. But, we've heard that it is challenging to get started. So, we are launching a series of short videos focused on explaining how to perform evaluations using LangSmith.

This video focuses on repetitions, which allows you to run multiple trials of an experiment. This can be very useful to build confidence in a result, especially in cases where the app being evaluator or evaluator itself have some variability in their behavior.

Docs:
https://docs.smith.langchain.com/how_to_guides/evaluation/evaluate_llm_application#evaluate-on-a-dataset-with-repetitions