text
🔥 Prometheus 2 was recently released by Kaist AI as an alternative and closely mirroring both human and GPT-4 evaluation, and surpassing the former Prometheus!
🔥 Prometheus 2 was recently released by Kaist AI as an alternative and closely mirroring both human and GPT-4 evaluation, and surpassing the former Prometheus!
new_line
new_line
resource
https://huggingface.co/prometheus-eval/prometheus-7b-v2.0
model
prometheus-eval/prometheus-7b-v2.0
https://huggingface.co/prometheus-eval/prometheus-7b-v2.0
new_line
resource
https://huggingface.co/prometheus-eval/prometheus-8x7b-v2.0
model
prometheus-eval/prometheus-8x7b-v2.0
https://huggingface.co/prometheus-eval/prometheus-8x7b-v2.0
new_line
new_line
text
🌬️Fine-tuned on top of 
🌬️Fine-tuned on top of 
resource
https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2
model
mistralai/Mistral-7B-Instruct-v0.2
https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2
text
 and 
 and 
resource
https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1
model
mistralai/Mixtral-8x7B-Instruct-v0.1
https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1
new_line
text
🗂️The datasets used for fine-tuning have been publicly released i.e. 
🗂️The datasets used for fine-tuning have been publicly released i.e. 
resource
https://huggingface.co/datasets/prometheus-eval/Feedback-Collection
dataset
prometheus-eval/Feedback-Collection
https://huggingface.co/datasets/prometheus-eval/Feedback-Collection
text
 and 
 and 
resource
https://huggingface.co/datasets/prometheus-eval/Preference-Collection
dataset
prometheus-eval/Preference-Collection
https://huggingface.co/datasets/prometheus-eval/Preference-Collection
new_line
text
🤝🏻Unified LM evaluator for absolute (a single prompt-completion pair) and relative (two completions for a given prompt) due to model merging
🤝🏻Unified LM evaluator for absolute (a single prompt-completion pair) and relative (two completions for a given prompt) due to model merging
new_line
text
❌No longer needs a mandatory reference / golden answer, but can still be provided optionally
❌No longer needs a mandatory reference / golden answer, but can still be provided optionally
new_line
text
🔝Surpasses the former version of Prometheus, and has a high correlation with human, GPT-4, and Claude 3 Opus scores when evaluating LMs
🔝Surpasses the former version of Prometheus, and has a high correlation with human, GPT-4, and Claude 3 Opus scores when evaluating LMs
new_line
text
📝Apache 2.0 license
📝Apache 2.0 license
new_line
new_line
text
Long-story short, an amazing job from Kaist AI bridging the gap with LLM evaluators other than proprietary and bigger models!
Long-story short, an amazing job from Kaist AI bridging the gap with LLM evaluators other than proprietary and bigger models!
new_line
new_line
text
This week at Argilla, we decided to add a new task to use Prometheus 2 as an LLM evaluator using 
This week at Argilla, we decided to add a new task to use Prometheus 2 as an LLM evaluator using 
inline_code
`distilabel`
distilabel
text
, so we implemented 
, so we implemented 
inline_code
`PrometheusEval`
PrometheusEval
text
.
.
new_line
new_line
text
😱 Using 
😱 Using 
inline_code
`PrometheusEval`
PrometheusEval
text
 running their 7B variant with vLLM in a single L40 on top of 
 running their 7B variant with vLLM in a single L40 on top of 
resource
https://huggingface.co/datasets/HuggingFaceH4/instruction-dataset
dataset
HuggingFaceH4/instruction-dataset
https://huggingface.co/datasets/HuggingFaceH4/instruction-dataset
text
, we got the 327 existing prompt-completion pairs evaluated and pushed to the Hub in less than 2 minutes!
, we got the 327 existing prompt-completion pairs evaluated and pushed to the Hub in less than 2 minutes!
new_line
new_line
text
Find the generated dataset and the code at 
Find the generated dataset and the code at 
resource
https://huggingface.co/datasets/distilabel-internal-testing/instruction-dataset-prometheus
dataset
distilabel-internal-testing/instruction-dataset-prometheus
https://huggingface.co/datasets/distilabel-internal-testing/instruction-dataset-prometheus
🔥 Prometheus 2 was recently released by Kaist AI as an alternative and closely mirroring both human and GPT-4 evaluation, and surpassing the former Prometheus!

https://huggingface.co/prometheus-eval/prometheus-7b-v2.0
https://huggingface.co/prometheus-eval/prometheus-8x7b-v2.0

🌬️Fine-tuned on top of https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2 and https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1
🗂️The datasets used for fine-tuning have been publicly released i.e. https://huggingface.co/datasets/prometheus-eval/Feedback-Collection and https://huggingface.co/datasets/prometheus-eval/Preference-Collection
🤝🏻Unified LM evaluator for absolute (a single prompt-completion pair) and relative (two completions for a given prompt) due to model merging
❌No longer needs a mandatory reference / golden answer, but can still be provided optionally
🔝Surpasses the former version of Prometheus, and has a high correlation with human, GPT-4, and Claude 3 Opus scores when evaluating LMs
📝Apache 2.0 license

Long-story short, an amazing job from Kaist AI bridging the gap with LLM evaluators other than proprietary and bigger models!

This week at Argilla, we decided to add a new task to use Prometheus 2 as an LLM evaluator using `distilabel`, so we implemented `PrometheusEval`.

😱 Using `PrometheusEval` running their 7B variant with vLLM in a single L40 on top of https://huggingface.co/datasets/HuggingFaceH4/instruction-dataset, we got the 327 existing prompt-completion pairs evaluated and pushed to the Hub in less than 2 minutes!

Find the generated dataset and the code at https://huggingface.co/datasets/distilabel-internal-testing/instruction-dataset-prometheus
🔥
matinla lunarflu davanstrien andrewrreed victor NickyNicky QuocKhanh osanseviero dhruva-sarma shamy777
❤️
matinla lunarflu andrewrreed QuocKhanh osanseviero anakin87
/avatars/5ffbcc32035ad5a5c83e45a4910c0541.svg
matinlaghaei
matinla
user