How to compute per-token loss when doing language modeling?
I’m fine-tuning GPT-2 on a language modeling task. Given a sequence, I’d like to compute the per-token loss for each token (instead of the averaged loss over the sequence). How can I do it?
2023-06-28T19:05:58Z
Hey, it’s been some time since you posted this. Were you able to do it? If yes, would you be able to share code for it?
2023-08-23T15:21:48Z
In pytorch, you can useCrossEntropyLossfunction withreductionparam set tonone(CrossEntropyLoss — PyTorch 2.0 documentation).You can find an example in a HF course:Training a causal language model from scratch - Hugging Face NLP Course(search forkeytoken_weighted_lossto find it)
2023-08-23T16:00:41Z
Thanks for your reply@chrisociepa. Yes, I was able to work it withreduction='none'