Can you provide an example of best practices for incorporating a pretrained HuggingFace Vision Transformer (ViT) into a PyTorch Lightning module?
While there are numerous examples andnotebooks showinghow to run and fine-tune pretrained models like Vision Transformers (ViT), I’m looking for a clear example of how to integrate a pretrained ViT into a PyTorch Lightning pipeline. Specifically:Should I instantiate theAutoImageProcessor()class within mypl.LightningModule, or would it be better to do so in mypl.DataModule?Should I implement my ownforwardmethod in theLightningModule, or should I simply call theforwardmethod of the pretrained model (which would be an attribute of my Lightning class)?