text
Although this might sound like another way to make money on LLM API calls...
Although this might sound like another way to make money on LLM API calls...
new_line
new_line
text
Good folks at 
Good folks at 
mention
@AnthropicAI
AnthropicAI
text
 just introduced Contextual Retrieval, and it's a significant yet logical step up from simple Retrieval-Augmented Generation (RAG)!
 just introduced Contextual Retrieval, and it's a significant yet logical step up from simple Retrieval-Augmented Generation (RAG)!
new_line
new_line
text
Here are the steps to implement Contextual Retrieval based on Anthropic's approach:
Here are the steps to implement Contextual Retrieval based on Anthropic's approach:
new_line
new_line
text
1. Preprocess the knowledge base:
1. Preprocess the knowledge base:
new_line
text
   - Break down documents into smaller chunks (typically a few hundred tokens each).
   - Break down documents into smaller chunks (typically a few hundred tokens each).
new_line
text
   - Generate contextual information for each chunk using Claude 3 Haiku with a specific prompt.
   - Generate contextual information for each chunk using Claude 3 Haiku with a specific prompt.
new_line
text
   - Prepend the generated context (usually 50-100 tokens) to each chunk.
   - Prepend the generated context (usually 50-100 tokens) to each chunk.
new_line
new_line
text
2. Create embeddings and a BM25 index:
2. Create embeddings and a BM25 index:
new_line
text
   - Use an embedding model (Gemini or Voyage recommended) to convert contextualized chunks into vector embeddings.
   - Use an embedding model (Gemini or Voyage recommended) to convert contextualized chunks into vector embeddings.
new_line
text
   - Create a BM25 index using the contextualized chunks.
   - Create a BM25 index using the contextualized chunks.
new_line
new_line
text
3. Set up the retrieval process:
3. Set up the retrieval process:
new_line
text
   - Implement a system to search both the vector embeddings and the BM25 index.
   - Implement a system to search both the vector embeddings and the BM25 index.
new_line
text
   - Use rank fusion techniques to combine and deduplicate results from both searches.
   - Use rank fusion techniques to combine and deduplicate results from both searches.
new_line
new_line
text
4. Implement reranking (optional but recommended):
4. Implement reranking (optional but recommended):
new_line
text
   - Retrieve the top 150 potentially relevant chunks initially.
   - Retrieve the top 150 potentially relevant chunks initially.
new_line
text
   - Use a reranking model (e.g., Cohere reranker) to score these chunks based on relevance to the query.
   - Use a reranking model (e.g., Cohere reranker) to score these chunks based on relevance to the query.
new_line
text
   - Select the top 20 chunks after reranking.
   - Select the top 20 chunks after reranking.
new_line
new_line
text
5. Integrate with the generative model:
5. Integrate with the generative model:
new_line
text
   - Add the top 20 chunks (or top K, based on your specific needs) to the prompt sent to the generative model.
   - Add the top 20 chunks (or top K, based on your specific needs) to the prompt sent to the generative model.
new_line
new_line
text
6. Optimize for your use case:
6. Optimize for your use case:
new_line
text
   - Experiment with chunk sizes, boundary selection, and overlap.
   - Experiment with chunk sizes, boundary selection, and overlap.
new_line
text
   - Consider creating custom contextualizer prompts for your specific domain.
   - Consider creating custom contextualizer prompts for your specific domain.
new_line
text
   - Test different numbers of retrieved chunks (5, 10, 20) to find the optimal balance.
   - Test different numbers of retrieved chunks (5, 10, 20) to find the optimal balance.
new_line
new_line
text
7. Leverage prompt caching:
7. Leverage prompt caching:
new_line
text
   - Use Claude's prompt caching feature to reduce costs when generating contextualized chunks.
   - Use Claude's prompt caching feature to reduce costs when generating contextualized chunks.
new_line
text
   - Cache the reference document once and reference it for each chunk, rather than passing it repeatedly.
   - Cache the reference document once and reference it for each chunk, rather than passing it repeatedly.
new_line
new_line
text
8. Evaluate and iterate
8. Evaluate and iterate
Although this might sound like another way to make money on LLM API calls...

Good folks at @AnthropicAI just introduced Contextual Retrieval, and it's a significant yet logical step up from simple Retrieval-Augmented Generation (RAG)!

Here are the steps to implement Contextual Retrieval based on Anthropic's approach:

1. Preprocess the knowledge base:
   - Break down documents into smaller chunks (typically a few hundred tokens each).
   - Generate contextual information for each chunk using Claude 3 Haiku with a specific prompt.
   - Prepend the generated context (usually 50-100 tokens) to each chunk.

2. Create embeddings and a BM25 index:
   - Use an embedding model (Gemini or Voyage recommended) to convert contextualized chunks into vector embeddings.
   - Create a BM25 index using the contextualized chunks.

3. Set up the retrieval process:
   - Implement a system to search both the vector embeddings and the BM25 index.
   - Use rank fusion techniques to combine and deduplicate results from both searches.

4. Implement reranking (optional but recommended):
   - Retrieve the top 150 potentially relevant chunks initially.
   - Use a reranking model (e.g., Cohere reranker) to score these chunks based on relevance to the query.
   - Select the top 20 chunks after reranking.

5. Integrate with the generative model:
   - Add the top 20 chunks (or top K, based on your specific needs) to the prompt sent to the generative model.

6. Optimize for your use case:
   - Experiment with chunk sizes, boundary selection, and overlap.
   - Consider creating custom contextualizer prompts for your specific domain.
   - Test different numbers of retrieved chunks (5, 10, 20) to find the optimal balance.

7. Leverage prompt caching:
   - Use Claude's prompt caching feature to reduce costs when generating contextualized chunks.
   - Cache the reference document once and reference it for each chunk, rather than passing it repeatedly.

8. Evaluate and iterate
👀
John6666 KingNish clem victor louisbrulenaudet
🔥
badamczewski01 mathiasn1