Vector search from text-image pairs : separate or common space?
Hi,I have 15M (text, image) product pairs and this volume doubles every 2 months. Given an entry pair (text, image), I need to find the most similar pairs.Text length varies from 1 to 20 words, image quality varies from 1 to 10. All combinations of good/poor text/image are possibleSimilarity = cosine , storage = ChromaTraining and inference budgets are lowThis problem is quiet common (seethis post) and I’m investigating several options:Cascade filtering from separate embeddings: find most similar texts, then filter most similar images or vice-versaStacked separate embeddingsv_image_i = ViT-B-32.encode(image)v_text_i = LLM.encode(text)similarity(i,j) = cosine(np.stack(v_image_i, v_text_i), np.stack(v_image_j, v_text_j))Unique latent-spaceWith any VisionTextDualEncoder from HF: similarity(i,j) = cosine(v_latent_i, v_latent_j)ASIF common space from top frozen encoders ([2210.01738] ASIF: Coupled Data Turns Unimodal Models to Multimodal Without Training)An intuition is that a unique latent-space will capture less fine-grained similarity than separate embeddings.Given the state-of-art in July 2023, what do you think is the best option? Could anyone share some benchmarks / insights about expected performance?Thank youTom