text
 🎈 LLM Benchmarks Update!
 🎈 LLM Benchmarks Update!
new_line
new_line
text
**tl;dr: do not depend on benchmark leaderboards to choose your "chatbot" model! (Especially for non-English languages.)**
**tl;dr: do not depend on benchmark leaderboards to choose your "chatbot" model! (Especially for non-English languages.)**
new_line
new_line
text
First of all, I'm discontinuing the Open #Dutch #LLM Leaderboard (
First of all, I'm discontinuing the Open #Dutch #LLM Leaderboard (
link
https://lnkd.in/eFnsaFR6
https://lnkd.in/eFnsaFR6
text
). It will stay online for now, but I urge the use of the ScandEval leaderboard instead (
). It will stay online for now, but I urge the use of the ScandEval leaderboard instead (
link
https://scandeval.com/dutch-nlg/
https://scandeval.com/dutch-nlg/
text
) by 
) by 
mention
@saattrupdan
saattrupdan
text
. It contains more tasks, has better reproducibility and statistics (CI) and a flexible back-end library (
. It contains more tasks, has better reproducibility and statistics (CI) and a flexible back-end library (
inline_code
`scandeval`
scandeval
text
) to run your own benchmarks with. As part of project "Leesplank" (with Michiel Buisman and Maarten Lens-FitzGerald) we recently added GPT-4-1106-preview scores to add a good "target" to the leaderboard.
) to run your own benchmarks with. As part of project "Leesplank" (with Michiel Buisman and Maarten Lens-FitzGerald) we recently added GPT-4-1106-preview scores to add a good "target" to the leaderboard.
new_line
new_line
text
An important note here is that benchmark leaderboards are not a golden truth. Especially evaluating generative models is hard. You run into issues like prompt engineering (and sensitivity of models to one or other prompt), structured output generation, and - quite simply - "how to automatically evaluate open-ended generation".
An important note here is that benchmark leaderboards are not a golden truth. Especially evaluating generative models is hard. You run into issues like prompt engineering (and sensitivity of models to one or other prompt), structured output generation, and - quite simply - "how to automatically evaluate open-ended generation".
new_line
new_line
text
💡 Another important but under-discussed facet is the discrepancy between models' capability of understanding vs. generating *in different languages* (so the NLU part of NLG benchmarking). In other words: some of the listed models score really well on, e.g., MCQ benchmarks but are not suitable to use as DUTCH chat bots. Interestingly, some of these models seem to understand questions in Dutch and are able to pick the right answer (because they have good knowledge or reasoning skills), but generating fluent and grammatical Dutch is something else entirely! This is perhaps also true for humans: it's easier to sort-of grasp the meaning of a new language and answer with "Yes" or "No", but answering fluently in the language is much harder! Yet, your language production fluency does not necessarily say anything about your knowledge and reasoning skills.
💡 Another important but under-discussed facet is the discrepancy between models' capability of understanding vs. generating *in different languages* (so the NLU part of NLG benchmarking). In other words: some of the listed models score really well on, e.g., MCQ benchmarks but are not suitable to use as DUTCH chat bots. Interestingly, some of these models seem to understand questions in Dutch and are able to pick the right answer (because they have good knowledge or reasoning skills), but generating fluent and grammatical Dutch is something else entirely! This is perhaps also true for humans: it's easier to sort-of grasp the meaning of a new language and answer with "Yes" or "No", but answering fluently in the language is much harder! Yet, your language production fluency does not necessarily say anything about your knowledge and reasoning skills.
new_line
new_line
text
Hopefully we can get a chat arena for Dutch some day - user feedback is the most powerful metric!
Hopefully we can get a chat arena for Dutch some day - user feedback is the most powerful metric!
new_line
 🎈 LLM Benchmarks Update!

**tl;dr: do not depend on benchmark leaderboards to choose your "chatbot" model! (Especially for non-English languages.)**

First of all, I'm discontinuing the Open #Dutch #LLM Leaderboard (https://lnkd.in/eFnsaFR6). It will stay online for now, but I urge the use of the ScandEval leaderboard instead (https://scandeval.com/dutch-nlg/) by @saattrupdan. It contains more tasks, has better reproducibility and statistics (CI) and a flexible back-end library (`scandeval`) to run your own benchmarks with. As part of project "Leesplank" (with Michiel Buisman and Maarten Lens-FitzGerald) we recently added GPT-4-1106-preview scores to add a good "target" to the leaderboard.

An important note here is that benchmark leaderboards are not a golden truth. Especially evaluating generative models is hard. You run into issues like prompt engineering (and sensitivity of models to one or other prompt), structured output generation, and - quite simply - "how to automatically evaluate open-ended generation".

💡 Another important but under-discussed facet is the discrepancy between models' capability of understanding vs. generating *in different languages* (so the NLU part of NLG benchmarking). In other words: some of the listed models score really well on, e.g., MCQ benchmarks but are not suitable to use as DUTCH chat bots. Interestingly, some of these models seem to understand questions in Dutch and are able to pick the right answer (because they have good knowledge or reasoning skills), but generating fluent and grammatical Dutch is something else entirely! This is perhaps also true for humans: it's easier to sort-of grasp the meaning of a new language and answer with "Yes" or "No", but answering fluently in the language is much harder! Yet, your language production fluency does not necessarily say anything about your knowledge and reasoning skills.

Hopefully we can get a chat arena for Dutch some day - user feedback is the most powerful metric!

👍
davanstrien lunarflu nikgr saattrupdan reach-vb shiv2050 ssml2050
🤝
Rijgersberg osanseviero lunarflu robinsmits reach-vb
https://cdn-avatars.huggingface.co/v1/production/uploads/643033401572f43a4818cea3/CiAPL-E-6p-W8ilJ3t9vM.jpeg
Carlo Moro
cnmoro
user
/avatars/403b8f796fa7a149b131345c93117af4.svg
Robin Smits
robinsmits
user
https://cdn-avatars.huggingface.co/v1/production/uploads/1594192845975-5e1e17b6fcf41d740b6996a8.jpeg
Bram Vanroy
BramVanroy
user