How to find closest embedding vectors?
I have100Kknown embedding i.e.[emb_1, emb_2, ..., emb_100000]My task is given an embedding(embedding_new) find the closest 10 embedding from the above100kembedding.The way I am approaching this problem is brute force.Given a query to find the closest embeddings, I compareembedding_newwith[emb_1, emb_2, ..., emb_100000]and get the similarity score.Then I do quicksort of the similarity score to get the top10closest embedding.Is there a better way to achieve this?
2022-07-24T08:56:03Z
Hey there! I don’t think you can get much faster than this, but I may be wrong. Once you have the list of similarities, you can usetorch.topkto obtain the top results of interest.
2022-07-26T01:29:54Z
Yes, please have a look at Nearest Neighbour Search algorithms like Faiss, Hnsw, etc. You can also you libraries like annoy.Annoy Library:GitHub - spotify/annoy: Approximate Nearest Neighbors in C++/Python optimized for memory usage and loading/saving to diskThe above mentioned algorithms use special data structures like graphs/trees to reduce the run time from linear.Thanks,Sharan Babu