Is it possible to evaluate generations/output while fine-tuning a LLM?
Is it possible to run some prompts and generate outputs for these prompts during fine-tuning (with transformers.Trainer), e.g. on each eval step?I’ve seen some predictions in the w&b report for this blogherebut I am not sure whether these predictions were made by loading the respective checkpoints.
2023-04-03T13:43:02Z
If someone stumbles upon the same question:Hereis where to look to implement custom callbacks on evaluation.
2023-11-01T22:07:12Z
Late here, but Braintrust is a great tool / platform for evaluating your LLM. We have a simple library in Python/Typescript for running and logging evaluations so you can use our web UI to dig into the results.It’s free to use @https://braintrustdata.com/