Reusable custom heads, wrappers, and `from_pretrained`?
I have a (complicated) custom head type which I would like to test with multiple base models. I would like to achieve the following:Easily apply this custom head to different base modelsDefine custom functions (similar togenerate()) on this custom headBe able to dofrom_pretrained()to load the base model and randomly initialise the new headBe able to save and load the complete finetuned model with custom head using thefrom_pretrained()mechanismPoints 1 and 2 point me towards making a wrapper model.Points 3 and 4 point me towards inheriting from the base model.What’s the best practice in this situation? Is there some trick to make thefrom_pretrained()magic work with the wrapper approach?One option would be to make a function which automatically creates an inherited model given the base class, but that doesn’t seem like the neat thing to do.