text
🚨 We’ve just released a new tool to compare the performance of models in the 🤗 Open LLM Leaderboard: the Comparator 🎉
🚨 We’ve just released a new tool to compare the performance of models in the 🤗 Open LLM Leaderboard: the Comparator 🎉
new_line
resource
https://huggingface.co/spaces/open-llm-leaderboard/comparator
space
open-llm-leaderboard/comparator
https://huggingface.co/spaces/open-llm-leaderboard/comparator
new_line
new_line
text
Want to see how two different versions of LLaMA stack up? Let’s walk through a step-by-step comparison of LLaMA-3.1 and LLaMA-3.2. 🦙🧵👇
Want to see how two different versions of LLaMA stack up? Let’s walk through a step-by-step comparison of LLaMA-3.1 and LLaMA-3.2. 🦙🧵👇
new_line
new_line
text
1/ Load the Models' Results
1/ Load the Models' Results
new_line
text
- Go to the 🤗 Open LLM Leaderboard Comparator: 
- Go to the 🤗 Open LLM Leaderboard Comparator: 
resource
https://huggingface.co/spaces/open-llm-leaderboard/comparator
space
open-llm-leaderboard/comparator
https://huggingface.co/spaces/open-llm-leaderboard/comparator
new_line
text
- Search for "LLaMA-3.1" and "LLaMA-3.2" in the model dropdowns.
- Search for "LLaMA-3.1" and "LLaMA-3.2" in the model dropdowns.
new_line
text
- Press the Load button. Ready to dive into the results!
- Press the Load button. Ready to dive into the results!
new_line
new_line
text
2/ Compare Metric Results in the Results Tab 📊
2/ Compare Metric Results in the Results Tab 📊
new_line
text
- Head over to the Results tab.
- Head over to the Results tab.
new_line
text
- Here, you’ll see the performance metrics for each model, beautifully color-coded using a gradient to highlight performance differences: greener is better! 🌟
- Here, you’ll see the performance metrics for each model, beautifully color-coded using a gradient to highlight performance differences: greener is better! 🌟
new_line
text
- Want to focus on a specific task? Use the Task filter to hone in on comparisons for tasks like BBH or MMLU-Pro.
- Want to focus on a specific task? Use the Task filter to hone in on comparisons for tasks like BBH or MMLU-Pro.
new_line
new_line
text
3/ Check Config Alignment in the Configs Tab ⚙️
3/ Check Config Alignment in the Configs Tab ⚙️
new_line
text
- To ensure you’re comparing apples to apples, head to the Configs tab.
- To ensure you’re comparing apples to apples, head to the Configs tab.
new_line
text
- Review both models’ evaluation configurations, such as metrics, datasets, prompts, few-shot configs...
- Review both models’ evaluation configurations, such as metrics, datasets, prompts, few-shot configs...
new_line
text
- If something looks off, it’s good to know before drawing conclusions! ✅
- If something looks off, it’s good to know before drawing conclusions! ✅
new_line
new_line
text
4/ Compare Predictions by Sample in the Details Tab 🔍
4/ Compare Predictions by Sample in the Details Tab 🔍
new_line
text
- Curious about how each model responds to specific inputs? The Details tab is your go-to!
- Curious about how each model responds to specific inputs? The Details tab is your go-to!
new_line
text
- Select a Task (e.g., MuSR) and then a Subtask (e.g., Murder Mystery) and then press the Load Details button.
- Select a Task (e.g., MuSR) and then a Subtask (e.g., Murder Mystery) and then press the Load Details button.
new_line
text
- Check out the side-by-side predictions and dive into the nuances of each model’s outputs.
- Check out the side-by-side predictions and dive into the nuances of each model’s outputs.
new_line
new_line
text
5/ With this tool, it’s never been easier to explore how small changes between model versions affect performance on a wide range of tasks. Whether you’re a researcher or enthusiast, you can instantly visualize improvements and dive into detailed comparisons.
5/ With this tool, it’s never been easier to explore how small changes between model versions affect performance on a wide range of tasks. Whether you’re a researcher or enthusiast, you can instantly visualize improvements and dive into detailed comparisons.
new_line
new_line
text
🚀 Try the 🤗 Open LLM Leaderboard Comparator now and take your model evaluations to the next level!
🚀 Try the 🤗 Open LLM Leaderboard Comparator now and take your model evaluations to the next level!
🚨 We’ve just released a new tool to compare the performance of models in the 🤗 Open LLM Leaderboard: the Comparator 🎉
https://huggingface.co/spaces/open-llm-leaderboard/comparator

Want to see how two different versions of LLaMA stack up? Let’s walk through a step-by-step comparison of LLaMA-3.1 and LLaMA-3.2. 🦙🧵👇

1/ Load the Models' Results
- Go to the 🤗 Open LLM Leaderboard Comparator: https://huggingface.co/spaces/open-llm-leaderboard/comparator
- Search for "LLaMA-3.1" and "LLaMA-3.2" in the model dropdowns.
- Press the Load button. Ready to dive into the results!

2/ Compare Metric Results in the Results Tab 📊
- Head over to the Results tab.
- Here, you’ll see the performance metrics for each model, beautifully color-coded using a gradient to highlight performance differences: greener is better! 🌟
- Want to focus on a specific task? Use the Task filter to hone in on comparisons for tasks like BBH or MMLU-Pro.

3/ Check Config Alignment in the Configs Tab ⚙️
- To ensure you’re comparing apples to apples, head to the Configs tab.
- Review both models’ evaluation configurations, such as metrics, datasets, prompts, few-shot configs...
- If something looks off, it’s good to know before drawing conclusions! ✅

4/ Compare Predictions by Sample in the Details Tab 🔍
- Curious about how each model responds to specific inputs? The Details tab is your go-to!
- Select a Task (e.g., MuSR) and then a Subtask (e.g., Murder Mystery) and then press the Load Details button.
- Check out the side-by-side predictions and dive into the nuances of each model’s outputs.

5/ With this tool, it’s never been easier to explore how small changes between model versions affect performance on a wide range of tasks. Whether you’re a researcher or enthusiast, you can instantly visualize improvements and dive into detailed comparisons.

🚀 Try the 🤗 Open LLM Leaderboard Comparator now and take your model evaluations to the next level!
👍
m4r14n3s akjindal53244 DmitryRyumin dvilasuero gabrielmbmb Joseph717171 louisbrulenaudet
👀
John6666 Joseph717171