How to evaluate before first training step?
Hi! I have a use case where I would like evaluation to happenat the beginning of training(before a training step has been taken) in addition to every n steps. I can easily get the latter usingevaluation_strategyandeval_steps, but not sure how to get the former.There is an argument,logging_first_stepthat sounds like it should doexactlywhat I need:logging_first_step(bool,optional, defaults toFalse) — Whether to log and evaluate the firstglobal_stepor not.But providing this argument does not lead to evaluation at the beginning of training like I would expect (tested with therun_summarization.pyscript). Does anyone have an idea how to get this behaviour from the HFTrainer?
2022-06-08T11:36:33Z
You can just add a linetrainer.evaluate()before the call totrain.
2022-06-08T16:14:58Z
Thanks for the quick response! I guess this solution works, but the results do not end up in thelog_historyfield of thetraining_state.jsonfile, which is how I am tracking performance over time. So I guess two questions:Is there a better way to track evaluation metrics over time (using the provided example scripts likerun_summarization.py) than thelog_historyfield of thetraining_state.json?Might adding this functionality (evaluating during trainingbeforea training step has been taken) be a good idea? Seems like a pretty ubiquitous use case to me as you may want to plot performance over time and knowing the performance of the model (either randomly initialized or pre-trained) before the first train step is useful. Would be happy to take a crack at this if I could get some advice as to where to implement it.
2023-01-17T14:41:19Z
A alternative approach, which may or may not solve the problem is to use a callback, like so:class EvaluateFirstStepCallback(TrainerCallback):
    def on_step_end(self, args, state, control, **kwargs):
        if state.global_step == 1:
            control.should_evaluate = True

trainer.add_callback(EvaluateFirstStepCallback())It does seem like it might be nice to have as a built-in TrainingArguments, given it mirrorslogging_first_steppretty closely.
2023-08-04T11:27:19Z
THanks for the post. One small issue here is, by doing this way, there will be one training step already done, which is not equal to the model evaluation without any training
2023-08-31T14:58:08Z
Doesn’t that get solved usingon_step_begin? Works for me for NLLB finetuning withSeq2SeqTrainingArguments. The only caveat seems to be that train loss doesn’t exist at this point so the wandb plots are offset.class EvaluateFirstStepCallback(TrainerCallback):
    def on_step_begin(self, args, state, control, **kwargs):
        if state.global_step == 1:
            control.should_evaluate = True

trainer.add_callback(EvaluateFirstStepCallback())
2023-09-23T19:41:20Z
The code provided by@frankierand@zouharviboth seem to have some minor errorsAccording totrainer.pyon_step_begin():if step % args.gradient_accumulation_steps == 0:, called before all operations except for the random state, whenglobal_stephas not yet been updated, and the model parameters have not yet been updatedon_step_end(): called after all operations in each training iteration, whenglobal_stephas been updated andwhenglobal_step == 1, if the gradient accumulation step is 1, the model parameters have been updated onceTherefore, it is necessary tocallon_step_begin()atglobal_step == 0in order to ensure that model parametersthat have not yet been updatedare evaluated.The corresponding code is below:class EvaluateFirstStepCallback(TrainerCallback):
    def on_step_begin(self, args, state, control, **kwargs):
        if state.global_step == 0:
            control.should_evaluate = True

trainer.add_callback(EvaluateFirstStepCallback())
2024-02-17T22:35:09Z
I wish they’d fix this in the actual code. It’s so annoying that the flag for logging delay doesn’t work as expected. Wasted 10-15 mins finding this solution.
2024-04-25T10:53:56Z
I have to say the progress bar will also step to “1” even using your providing code:图片2396×218 34.1 KB
2024-07-31T05:27:14Z
The reason is that evaluation happens after training, i.e. should_evaluate is only checked afterwards.
2024-08-03T16:12:12Z
Hey everybody!Now (2024) we can pass the parametereval_on_startto your initializedTrainingArgumentsobject to make your model evaluate before undergoing any training stepsThis new parameter was renamed from a deprecated one, “sanity_evaluation”, as introducedhere.I hope this update will find all of you well!