text
🎉 A new LLM is launched! 🚀 
🎉 A new LLM is launched! 🚀 
new_line
text
After checking if it's open-source or not, 🤔 
After checking if it's open-source or not, 🤔 
new_line
text
you rush to see the benchmarks... 🏃‍♂️💨
you rush to see the benchmarks... 🏃‍♂️💨
new_line
new_line
text
Which benchmark does everyone check first? 🔍
Which benchmark does everyone check first? 🔍
new_line
new_line
text
MMLU (Massive Multitask Language Understanding)? 📚
MMLU (Massive Multitask Language Understanding)? 📚
new_line
new_line
text
Benchmarks like MMLU reaching saturation... most of the time the performance does not translate to real-world use cases! 🌐❗
Benchmarks like MMLU reaching saturation... most of the time the performance does not translate to real-world use cases! 🌐❗
new_line
new_line
text
Meet MMLU-Pro, released by TIGER-Lab on 
Meet MMLU-Pro, released by TIGER-Lab on 
mention
@huggingface
huggingface
text
 ! 🐯🌍
 ! 🐯🌍
new_line
new_line
text
🧪 12,217 questions across biology, business, chemistry, computer science, economics, engineering, health, history, law, mathematics, philosophy, physics, and psychology carefully validated by humans 🧑‍🔬
🧪 12,217 questions across biology, business, chemistry, computer science, economics, engineering, health, history, law, mathematics, philosophy, physics, and psychology carefully validated by humans 🧑‍🔬
new_line
new_line
text
🔟 Goes to 10 options per question instead of 4, this increase in options will make the evaluation more realistic and reduce random guessing 🎯
🔟 Goes to 10 options per question instead of 4, this increase in options will make the evaluation more realistic and reduce random guessing 🎯
new_line
new_line
text
📊 56% of questions come from MMLU, 34% from STEM websites, and the rest from TheoremQA and SciBench 📈
📊 56% of questions come from MMLU, 34% from STEM websites, and the rest from TheoremQA and SciBench 📈
new_line
new_line
text
🤖 LLMs with weak chain-of-thought reasoning tend to perform lower, indicating it is more challenging and representative of real-world expectations 🧠💡
🤖 LLMs with weak chain-of-thought reasoning tend to perform lower, indicating it is more challenging and representative of real-world expectations 🧠💡
new_line
new_line
text
Any guess who tops it and who bombs it? 🤔📉📈
Any guess who tops it and who bombs it? 🤔📉📈
new_line
new_line
text
GPT-4o drops by 17% (from 0.887 to 0.7149) 📉
GPT-4o drops by 17% (from 0.887 to 0.7149) 📉
new_line
text
Llama-3-70B drops by 27% (from 0.820 to 0.5541) 📉
Llama-3-70B drops by 27% (from 0.820 to 0.5541) 📉
new_line
new_line
text
🔗 
🔗 
resource
https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro
dataset
TIGER-Lab/MMLU-Pro
https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro
🎉 A new LLM is launched! 🚀 
After checking if it's open-source or not, 🤔 
you rush to see the benchmarks... 🏃‍♂️💨

Which benchmark does everyone check first? 🔍

MMLU (Massive Multitask Language Understanding)? 📚

Benchmarks like MMLU reaching saturation... most of the time the performance does not translate to real-world use cases! 🌐❗

Meet MMLU-Pro, released by TIGER-Lab on @huggingface ! 🐯🌍

🧪 12,217 questions across biology, business, chemistry, computer science, economics, engineering, health, history, law, mathematics, philosophy, physics, and psychology carefully validated by humans 🧑‍🔬

🔟 Goes to 10 options per question instead of 4, this increase in options will make the evaluation more realistic and reduce random guessing 🎯

📊 56% of questions come from MMLU, 34% from STEM websites, and the rest from TheoremQA and SciBench 📈

🤖 LLMs with weak chain-of-thought reasoning tend to perform lower, indicating it is more challenging and representative of real-world expectations 🧠💡

Any guess who tops it and who bombs it? 🤔📉📈

GPT-4o drops by 17% (from 0.887 to 0.7149) 📉
Llama-3-70B drops by 27% (from 0.820 to 0.5541) 📉

🔗 https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro
🔥
clem zaursamedov1 osanseviero
❤️
clem zaursamedov1 clefourrier
https://cdn-avatars.huggingface.co/v1/production/uploads/1583857146757-5e67bdd61009063689407479.jpeg
Clem 🤗
clem
user
/avatars/e4fa0d942a2e23bffb365601a34c4c9d.svg
Z
Zachyyypoo
user