How to use ViT MAE for image classification?
My use case it to use the pre-trained weights of MAE to do image classification on my custom image classification datasets.For the ViT model, there are implementations for bothViTModel(which returns raw hidden-states without any specific head on top) andViTForImageClassification(which adds a linear layer on top of the final hidden state of the [CLS] token).MAE has been considered as the state-of-the-art image classification model among models without external data. However, it turns out that Hugging Face only providesViTMAEModelwhich returns raw hidden-states without any specific head on top, it does not provideViTMAEForImageClassificationnorViTMAEModelForImageClassification. Is there any particular reason of not doing it?One explanation I could think of is that MAE shares the same model architecture as ViT, so one could do the following to fulfil my use case:image_processor = AutoImageProcessor.from_pretrained("facebook/vit-mae-base")
model = ViTForImageClassification.from_pretrained("facebook/vit-mae-base")Is the above code the correct way to use the pre-trained weights of MAE to do image classification?If yes, I wonder what the intended use cases ofViTMAEModelare?For the MAE pretraining, we already haveViTMAEForPreTraining, soViTMAEModelis not used for pre-training.@nielsr@sayakpaulCould you please help me?
2023-04-04T04:25:33Z
MAE has been considered as the state-of-the-art image classification model among models without external data. However, it turns out that Hugging Face only providesViTMAEModelwhich returns raw hidden-states without any specific head on top, it does not provideViTMAEForImageClassificationnorViTMAEModelForImageClassification. Is there any particular reason of not doing it?It’s likely because when the model was open-sourced, it didn’t have the classification head params.One explanation I could think of is that MAE shares the same model architecture as ViT, so one could do the following to fulfil my use case:If you do the above, then the classification head parameters will be randomly initialized which is likely what you want when fine-tuning.If yes, I wonder what the intended use cases ofViTMAEModelare?It’s useful for feature extraction. See this notebook where I used a similar model class called:TFData2VecVisionModel:https://github.com/sayakpaul/TF-2.0-Hacks/blob/master/data2vec_vision_image_classification.ipynb
2023-04-04T07:54:46Z
Thanks a lot for your reply!It’s useful for feature extraction. See this notebook where I used a similar model class called:TFData2VecVisionModel:TF-2.0-Hacks/data2vec_vision_image_classification.ipynb at master · sayakpaul/TF-2.0-Hacks · GitHubFor this use case (feature extraction), I am wondering it is sufficient to useViTModelas shown below, is it correct?If yes, thenViTMAEModelstill does not have its own unique use case, but it can be see as a utility class to buildViTMAEForPreTraining.model = ViTModel.from_pretrained("facebook/vit-mae-base")
2024-09-04T20:02:38Z
Hello!  Do you have any update about your idea?