Use Trainer API with two valiation sets
Hi everyone,right now theTrainerAPI accepts oneeval_dataset. I am wondering, is it somehow possible to provide two different validation sets that are both evaluated during training? For example, I might want to track my validation loss on a validation set that was previously sampled from my training data and hence shares the same distribution and on a validation set that was sampled from a data set with a presumably different data distribution (e.g., stemming from a different period). The idea stems from“Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks”.Thanks in advanceSimon
2022-02-28T12:11:55Z
@patrickvonplatenthis a common use case in research (including in mine). I don’t see any examples in the docs, but the docs (Callbacks) suggest that  there should be a way to write a callback function to achieve this? Looking at the code very briefly, I imagine that a callback would simply call theevaluateagain for all othereval_datasets at the end of the validation loop, optionally changingmetric_key_prefixso that the logger displays traces of the metrics measured on individual datasets separately?I’d have to look in more detail at the APIs for implementing a callback to provide more details, but I’m thinking.UPDATE: I had a look but I can’t see how I could pass the other datasets to the callback, so I assume this is not possible. To promote a productive discussion, I raised a feature request where I summarise the issue and a potential workaroundhere.