text
Every time a new model is released that is topping 10+ leaderboards on 50+ benchmarks... 🚀
Every time a new model is released that is topping 10+ leaderboards on 50+ benchmarks... 🚀
new_line
new_line
text
My brain goes... I will wait for the LMSYS Chatbot Arena results! 🤔
My brain goes... I will wait for the LMSYS Chatbot Arena results! 🤔
new_line
new_line
text
User-facing evaluation, such as Chatbot Arena, provides reliable signals but is costly and slow. 🐢
User-facing evaluation, such as Chatbot Arena, provides reliable signals but is costly and slow. 🐢
new_line
new_line
text
Now we have MixEval, a new open benchmark with a 96% correlation to LMSYS Chatbot Arena and Human preferences. 🎯
Now we have MixEval, a new open benchmark with a 96% correlation to LMSYS Chatbot Arena and Human preferences. 🎯
new_line
new_line
text
It comes with MixEval (4k samples) and MixEval Hard (1k samples) 📊
It comes with MixEval (4k samples) and MixEval Hard (1k samples) 📊
new_line
new_line
text
Can use GPT-3.5-Turbo or any other open-source models as Parser/Judge 🤖
Can use GPT-3.5-Turbo or any other open-source models as Parser/Judge 🤖
new_line
new_line
text
It takes less than 6% of the time and cost of MMLU 💸
It takes less than 6% of the time and cost of MMLU 💸
new_line
new_line
text
As expected:
As expected:
new_line
text
In open models: Qwen2 72B >> Llama 3 70B >> Mixtral 8x7B 🔝
In open models: Qwen2 72B >> Llama 3 70B >> Mixtral 8x7B 🔝
new_line
text
In Closed Models: GPT-4o >> Claude 3 Opus >> Gemini Pro 🔒
In Closed Models: GPT-4o >> Claude 3 Opus >> Gemini Pro 🔒
new_line
new_line
text
Leaderboard: 
Leaderboard: 
link
https://mixeval.github.io/
https://mixeval.github.io/
text
  📈
  📈
Every time a new model is released that is topping 10+ leaderboards on 50+ benchmarks... 🚀

My brain goes... I will wait for the LMSYS Chatbot Arena results! 🤔

User-facing evaluation, such as Chatbot Arena, provides reliable signals but is costly and slow. 🐢

Now we have MixEval, a new open benchmark with a 96% correlation to LMSYS Chatbot Arena and Human preferences. 🎯

It comes with MixEval (4k samples) and MixEval Hard (1k samples) 📊

Can use GPT-3.5-Turbo or any other open-source models as Parser/Judge 🤖

It takes less than 6% of the time and cost of MMLU 💸

As expected:
In open models: Qwen2 72B >> Llama 3 70B >> Mixtral 8x7B 🔝
In Closed Models: GPT-4o >> Claude 3 Opus >> Gemini Pro 🔒

Leaderboard: https://mixeval.github.io/  📈
👀
Smooke GPT007 Rybens alielfilali01