 Okay, after fixing the max steps parameter, our training run has now finished. And we can see the training loss has converged over a little bit more than a thousand steps. And we can also see the generations from our model that we logged in evaluation. We can see both the prompt and the generation. But it would be good now to compare this versus the baseline and see how this looks so now instead of running evaluation manually i will go to artifacts i will go to the model that we saved in this training run and i will link it to registry and remember we're working with a small instructor lm and now i will specifically add the candidate alias to this model version and after linking it we should be able to see um to see this model in the registry and hopefully the evaluation action has triggered. So let's take a look at the queue. And you can see the status for a new run is queued. It's waiting for an agent, and it's running now, so the agent has picked up this specific evaluation run. And we should be able to go there. And we can double check the config. So in the config, we can see that the candidate model is now small instruct lm v3. And it's evaluated against our baseline which is the version zero and we can see when this run finishes we should be able to see the the results it's uh still waiting for the i think the run is still is still going so let's wait for a little bit and then we'll see the results of this run okay we can see now the candidate preference score is 0.53 so now this this run is a lot better than than our baseline so this is a candidate that based on this based on this score we can do something more with it maybe we can move it to the next stage maybe we can move it to production maybe we can move it to the the stage of quantization we should be able to see the comparison between the baseline and the candidate. And we can, for example, filter by the choice and, sorry, group by the choice and see that there's just 12 examples where the baseline is better than our model. There are 23 cases when both are considered equal by our LLM judge, and in 65 cases the candidate is better. So it's clearly a better model than the baseline. If you want to do a deep dive and understand maybe where there might be regressions or where the candidate might be worse than the baseline, we can also use this table, maybe if I remove this, I remove this grouping, and just maybe I want to focus on the error analysis. And if I sort this in a distantly way, then these are all of the examples where the candidate is worse than the baseline. We can take a look maybe at some of the reasoning here from our llm judge um and use this as an insight to improving our model in the next run maybe we need to give it some more data maybe we need to change the hyper parameters and this will help us improve our model over time so hopefully you've seen the value of setting up all of this this automations and evaluation in an automated way based on actions from the model registry. And yeah, we're looking forward to having you practice this on your projects and in your organizations.
LLM Evaluation results
In this lesson we will take a look at the results from our automated evaluation runs and make conclusions about our candidate model.