Similarity search based on multiple text attributes
Good day expertsI need to build a system to recommend related rows.If query = row1, I need results ranked by similarity which combines signal from all attributes.Each row has multiple text attributes.Each of these attribute is a independent document.Attribute1 = doc1, Attribute2 = doc2… and so on.Number of documents per row is variableRow 1 may contain Attribute1 = doc1, Attribute 2 = doc2.Row 2 may contain Attribute1 = doc1, Attribute 2 = doc2 … Attribute10 = doc10.I considered two approaches.Concatenate first n characters of each attribute (doc) and get embeddings for one concatenated stringGet embeddings for each attribute (doc) separatelyApproach 1 saves us cost of fetching and storing embeddings. This requires concatenating chunks from separate documents. I have a concern. This approach may result in a concatenated string with abrupt change in content and semantic meaning at the points of concatenation.Approach 2 is costly, we need to fetch embeddings for multiple documents per row. However I think, the embeddings are more usable as each document retains its meaning.Question 1 - Which approach do experts recommend 1 or 2, and why ?Question 2 - Any guidance on doing similarity search for embeddings from multiple text attributes ?Can we simply concatenate the embeddings ?How do we handle rows with different number of attributes ie., different length of concatenated embeddings ?