text
Shocking: 2/3 of LLMs fail at 2K context length
Shocking: 2/3 of LLMs fail at 2K context length
new_line
new_line
text
code_your_own_ai makes a great vlog about mostly LLM related AI content. 
code_your_own_ai makes a great vlog about mostly LLM related AI content. 
new_line
text
As I watched the video below, I wondered about current best practices on LLM evaluation. We have benchmarks, we have sota LLMs evaluating LLMs, we have tools evaluating based on human comparison. 
As I watched the video below, I wondered about current best practices on LLM evaluation. We have benchmarks, we have sota LLMs evaluating LLMs, we have tools evaluating based on human comparison. 
new_line
text
Often, I hear, just play with the LLM for 15 mins to form an opinion. 
Often, I hear, just play with the LLM for 15 mins to form an opinion. 
new_line
text
While I think for a specific use case and clear expectations, this could yield signal carrying experiences, I also see that one prompt is used to judge models. 
While I think for a specific use case and clear expectations, this could yield signal carrying experiences, I also see that one prompt is used to judge models. 
new_line
text
While benchmarks have their weaknesses, and are by themselves not enough to judge model quality, I still think systematic methods that try to reduce various scientifically known errs should be the way forward, even for qualitative estimates. 
While benchmarks have their weaknesses, and are by themselves not enough to judge model quality, I still think systematic methods that try to reduce various scientifically known errs should be the way forward, even for qualitative estimates. 
new_line
text
What do you think? How can we make a public tool for judging models like lmsys/chatbot-arena-leaderboard help to leverage standards known in social science? 
What do you think? How can we make a public tool for judging models like lmsys/chatbot-arena-leaderboard help to leverage standards known in social science? 
new_line
new_line
link
https://www.youtube.com/watch?v=mWrivekFZMM
https://www.youtube.com/watch?v=mWrivekFZMM
Shocking: 2/3 of LLMs fail at 2K context length

code_your_own_ai makes a great vlog about mostly LLM related AI content. 
As I watched the video below, I wondered about current best practices on LLM evaluation. We have benchmarks, we have sota LLMs evaluating LLMs, we have tools evaluating based on human comparison. 
Often, I hear, just play with the LLM for 15 mins to form an opinion. 
While I think for a specific use case and clear expectations, this could yield signal carrying experiences, I also see that one prompt is used to judge models. 
While benchmarks have their weaknesses, and are by themselves not enough to judge model quality, I still think systematic methods that try to reduce various scientifically known errs should be the way forward, even for qualitative estimates. 
What do you think? How can we make a public tool for judging models like lmsys/chatbot-arena-leaderboard help to leverage standards known in social science? 

https://www.youtube.com/watch?v=mWrivekFZMM
🤯
clem osanseviero julien-c pierrci NameeO radames AdinaY Tanvir1337 ucalyptus frontenbrecher
👍
samusenps trtm
https://cdn-avatars.huggingface.co/v1/production/uploads/1673563939929-63c074053bdc86f81088aae2.png
Namee Oberst
NameeO
user
https://cdn-avatars.huggingface.co/v1/production/uploads/1669551186189-63732ebbbd81fae2b3aaf3fb.jpeg
Knut Jägersberg
KnutJaegersberg
user
https://cdn-avatars.huggingface.co/v1/production/uploads/630fdd96a119d49bc1e770d5/OpU95S4a8hkM8OUCZq79R.jpeg
Adam
adamo1139
user
https://cdn-avatars.huggingface.co/v1/production/uploads/62a3bb1cd0d8c2c2169f0b88/eT2TS0IlQbZtz-F_zHLz9.jpeg
Joseph [open/acc] Pollack
Tonic
user