CLIP-like models do not support .add_adapter method
When working with LoRA, it is common to use .add_adapter method to supply a particular layers of various NLP models such as DeBERTa, FLAN, T5, etc., with adapter inside those selected layers.This method differs from get_peft_model, because the first one do not create a wrapper outside basic the model which, sometimes, comes in handy.The problem is that this .add_adapter method is not supported for CLIP, ViT and other popular vision models.If I am wrong, please tell me.Best regards.
2024-03-10T18:58:20Z
Hi,The following works for me:from transformers import AutoModelForImageClassification
from peft import LoraConfig

config = LoraConfig(
    r=16,
    lora_alpha=16,
    target_modules=["query", "value"],
    lora_dropout=0.1,
    bias="none",
    modules_to_save=["classifier"],
)
model = AutoModelForImageClassification.from_pretrained("google/vit-base-patch16-224")

model.add_adapter(config)

for name, param in model.named_parameters():
    print(name, param.shape)This is thanks to the PEFT integration in Transformers:PEFT integrations.See also this tutorial:Image classification using LoRA.