text
🧪 RAG Evaluation with 🔥 Prometheus 2 + Haystack
🧪 RAG Evaluation with 🔥 Prometheus 2 + Haystack
new_line
new_line
text
📝 Blog post: 
📝 Blog post: 
link
https://haystack.deepset.ai/blog/rag-evaluation-with-prometheus-2
https://haystack.deepset.ai/blog/rag-evaluation-with-prometheus-2
new_line
text
📓 Notebook: 
📓 Notebook: 
link
https://github.com/deepset-ai/haystack-cookbook/blob/main/notebooks/prometheus2_evaluation.ipynb
https://github.com/deepset-ai/haystack-cookbook/blob/main/notebooks/prometheus2_evaluation.ipynb
new_line
new_line
text
─── ⋆⋅☆⋅⋆ ───
─── ⋆⋅☆⋅⋆ ───
new_line
new_line
text
When evaluating LLMs' responses, 𝐩𝐫𝐨𝐩𝐫𝐢𝐞𝐭𝐚𝐫𝐲 𝐦𝐨𝐝𝐞𝐥𝐬 like GPT-4  are commonly used due to their strong performance.
When evaluating LLMs' responses, 𝐩𝐫𝐨𝐩𝐫𝐢𝐞𝐭𝐚𝐫𝐲 𝐦𝐨𝐝𝐞𝐥𝐬 like GPT-4  are commonly used due to their strong performance.
new_line
text
However, relying on closed models presents challenges related to data privacy 🔒, transparency, controllability, and cost 💸.
However, relying on closed models presents challenges related to data privacy 🔒, transparency, controllability, and cost 💸.
new_line
new_line
text
On the other hand, 𝐨𝐩𝐞𝐧 𝐦𝐨𝐝𝐞𝐥𝐬 typically do not correlate well with human judgments and lack flexibility.
On the other hand, 𝐨𝐩𝐞𝐧 𝐦𝐨𝐝𝐞𝐥𝐬 typically do not correlate well with human judgments and lack flexibility.
new_line
new_line
new_line
text
🔥 Prometheus 2  is a new family of open-source models designed to address these gaps:
🔥 Prometheus 2  is a new family of open-source models designed to address these gaps:
new_line
text
🔹 two variants: 
🔹 two variants: 
resource
https://huggingface.co/prometheus-eval/prometheus-7b-v2.0
model
prometheus-eval/prometheus-7b-v2.0
https://huggingface.co/prometheus-eval/prometheus-7b-v2.0
text
; 
; 
resource
https://huggingface.co/prometheus-eval/prometheus-8x7b-v2.0
model
prometheus-eval/prometheus-8x7b-v2.0
https://huggingface.co/prometheus-eval/prometheus-8x7b-v2.0
new_line
text
🔹 trained on open-source data
🔹 trained on open-source data
new_line
text
🔹 high correlation with human evaluations and proprietary models
🔹 high correlation with human evaluations and proprietary models
new_line
text
🔹 highly flexible: capable of performing direct assessments and pairwise rankings, and allowing the definition of custom evaluation criteria.
🔹 highly flexible: capable of performing direct assessments and pairwise rankings, and allowing the definition of custom evaluation criteria.
new_line
new_line
text
See my experiments with RAG evaluation in the links above.
See my experiments with RAG evaluation in the links above.
new_line
🧪 RAG Evaluation with 🔥 Prometheus 2 + Haystack

📝 Blog post: https://haystack.deepset.ai/blog/rag-evaluation-with-prometheus-2
📓 Notebook: https://github.com/deepset-ai/haystack-cookbook/blob/main/notebooks/prometheus2_evaluation.ipynb

─── ⋆⋅☆⋅⋆ ───

When evaluating LLMs' responses, 𝐩𝐫𝐨𝐩𝐫𝐢𝐞𝐭𝐚𝐫𝐲 𝐦𝐨𝐝𝐞𝐥𝐬 like GPT-4  are commonly used due to their strong performance.
However, relying on closed models presents challenges related to data privacy 🔒, transparency, controllability, and cost 💸.

On the other hand, 𝐨𝐩𝐞𝐧 𝐦𝐨𝐝𝐞𝐥𝐬 typically do not correlate well with human judgments and lack flexibility.


🔥 Prometheus 2  is a new family of open-source models designed to address these gaps:
🔹 two variants: https://huggingface.co/prometheus-eval/prometheus-7b-v2.0; https://huggingface.co/prometheus-eval/prometheus-8x7b-v2.0
🔹 trained on open-source data
🔹 high correlation with human evaluations and proprietary models
🔹 highly flexible: capable of performing direct assessments and pairwise rankings, and allowing the definition of custom evaluation criteria.

See my experiments with RAG evaluation in the links above.

🔥
leonardlin Marvin73