Stable Diffusion CLIP similarity
Hi,theStableDiffusionPipelineuses aCLIPTextModelfor the text embeddings, can I match that with the correspondingCLIPVisionModelto get CLIP test-image similarity of generated images? Which pretrainedCLIPVisionModelshould I use?Thanks!
2022-11-01T15:30:12Z
This is exactly what I am trying to do.But I am blocked by the fact that the CLIPVisionModel  embedding output has a different size than I expected.In thisminimum working exampleyou can see that despiteCLIPVisionModel.config.projection_dim == CLIPTextModel.config.projection_dim==768.Still, the embeddings present of different shapes:CLIPVisionEmbeddings(
  (patch_embedding): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14), bias=False)
  (position_embedding): Embedding(257, 1024)
)
CLIPTextEmbeddings(
  (token_embedding): Embedding(49408, 768)
  (position_embedding): Embedding(77, 768)
)I am confused about this and I can’t find much more in the documentation.
2022-11-01T16:50:23Z
Hello!You can compute the similarity score between text and image by computing the dot product between the corresponding (normalized) embeddings. In order to do so, you should use the same CLIP Vision model as the one used to retrieve the text embeddings. See for example this snippet here:transformers/modeling_clip.py at v4.24.0 · huggingface/transformers · GitHub. If you invoke that function with multiples images and texts,logits_per_imagewould give you the scores of each image across multiple text prompts, andlogits_per_textwould give you the score about how much each image matches the corresponding text. You can use one image and one prompt and get a score that would be comparable with other image-text pairs.There’s an usage example hereCLIPthat could be a good starting point
2022-11-01T17:19:06Z
Thanks!This is what I was looking for:with torch.no_grad():
    vision_outputs = model.vision_model(pixel_values=inputs.pixel_values)
    text_outputs = model.text_model(input_ids=inputs.input_ids)
    img_embeds = model.visual_projection(vision_outputs[1])
    txt_embeds = model.text_projection(text_outputs[1])
    img_embeds = img_embeds / img_embeds.norm(p=2, dim=-1, keepdim=True)
    txt_embeds = txt_embeds / txt_embeds.norm(p=2, dim=-1, keepdim=True)
img_embeds.shape, txt_embeds.shape(torch.Size([1, 768]), torch.Size([1, 768]))I was usingtransformers‘4.23.1’…
2022-11-01T17:27:17Z
Ok, I upgraded andmodel.vision_model.embeddings, model.text_model.embeddingsare not the same than projection embeddings. I guess I understood it wrongly, but anyway, I can do what I wanted with the projection.
2022-11-01T17:56:43Z
Oh yes, you are right, you need the projection embeddings
2022-12-06T17:07:46Z
Great that you solved it@fredguth! How did you initalize a working CLIP pipeline from the components of StableDiffusion?Thanks!