Track multiple losses & different outputs size with Trainer and callbacks
Hi,I’ve built a model that optimize jointly 2 different models and I would like to track these into Tensorboard or Wandb. Right now, I had to subclass the following methods: training_step (to return all losses and the one to optmize), train (to manage the new output from training_step and to add all losses into metrics).Is there a better way to integrate this logic (especially to not subclass train which is a very heavy method) ?Also, can Trainer manage different output size (ex: logits at token level and logits at sentence level) ? It seems that all logits must have the same shape currently, hence reshaping everything with the same dimension may lead to waste of memory.Thanks
2020-10-27T11:38:04Z
TheTrainerclass is not built to optimize two models at the same time, so no, there is no easier way than subclassing and overrifing thetraining_step. In general, subclassing the Trainer and overriding the method(s) to fit your needs is the expected way and we designed the Trainer API to make it as easy as possible.For predict/evaluate, yes Trainer will need tensors of the same size (with the exception of the batch dimension) otherwise it won’t be able to concatenate all predictions. This is something we’ll look into more when we rewrite the token-classification examples (in the next few weeks).
2020-10-27T19:12:28Z
Thanks a lot for the confirmation
2022-03-15T03:44:43Z
Hi, what if I am doing MTL (multi-task-learning) and just want to output all task losses? What’s the easiest way to do so?
2024-07-11T02:11:36Z
I implemented a way to track multiple losses:hf-multitask-trainer. it’s available for hf trainer.