CLIPTextModel's get_text_features VS pooled outputs
Hi all,I am trying to obtain embedding vectors for text using CLIP that will eventually be used for cross-attention with a latent diffusion model (the standard text-to-image approach). For context, the embedding vectors will first pass through some of my own linear projection layers during training before being fed into the UNet.A common practice for obtaining CLIP text embeddings is to useCLIPTextModel’sget_text_featuresmethod. From thecode, I can see that it’s projecting the pooled outputs using a linearself.text_projectionlayer. I have also verified that this is true by comparing against the.pooler_outputfromCLIPTextModel.My question is, what’s thisself.text_projectionfor specifically? Its projecting from 768 to 768 so dimensionality doesn’t change. Does thetext_projectionmatter at all in my case if I can just use the pooled outputs directly fromCLIPTextModel? I also observe people using pooled outputs for obtaining CLIP’s text embeddings so it’s confusing when to do what here.Apologies if this question might be a bit nuanced.
2024-08-30T10:34:51Z
I think the projection layer is just there to project the text features (coming from the text encoder) and the image features (coming from the vision encoder) into the same embedding space. The vision encoder uses a different dimensionality (e.g. 1024 foropenai/clip-vit-large-patch14 · Hugging Face), so the projection layers on both the text and vision sides make sure this is projected to the same dimensionality (like 512 or 768).