text
As part of the Data is Better Together MPEP project, we are now at the point where some translation efforts have successfully translated 500 highly ranked prompts into a new target language (amazing work from 
As part of the Data is Better Together MPEP project, we are now at the point where some translation efforts have successfully translated 500 highly ranked prompts into a new target language (amazing work from 
mention
@Rijgersberg
Rijgersberg
text
 et al!)
 et al!)
new_line
new_line
text
Our next step is to use these translated prompts to evaluate the performance of LLMs for non English languages. 
Our next step is to use these translated prompts to evaluate the performance of LLMs for non English languages. 
new_line
new_line
text
Does LLM, as a judge, work outside of English?
Does LLM, as a judge, work outside of English?
new_line
new_line
text
Ideally, it would be compelling to leverage LLMs to judge models for non-English since this significantly lowers the barrier to evaluating models (although it doesn't remove this barrier altogether). 
Ideally, it would be compelling to leverage LLMs to judge models for non-English since this significantly lowers the barrier to evaluating models (although it doesn't remove this barrier altogether). 
new_line
new_line
text
What we want to know is:
What we want to know is:
new_line
text
- does auto/LLM eval work in general for a particular language
- does auto/LLM eval work in general for a particular language
new_line
text
- which model(s) works best as a judge
- which model(s) works best as a judge
new_line
text
- do LLMs' judgments of non-English models match human preferences? 
- do LLMs' judgments of non-English models match human preferences? 
new_line
new_line
text
We're starting to think about how to approach this. If you have any ideas of possible approaches feel free to comment or join the discussion here: 
We're starting to think about how to approach this. If you have any ideas of possible approaches feel free to comment or join the discussion here: 
link
https://github.com/huggingface/data-is-better-together/issues/61
https://github.com/huggingface/data-is-better-together/issues/61
text
new_line
new_line
text
Other ideas...
Other ideas...
new_line
new_line
text
Could an approach like 
Could an approach like 
resource
https://huggingface.co/papers/2404.18796
paper
2404.18796
https://huggingface.co/papers/2404.18796
Replacing Judges with Juries: Evaluating LLM Generations with a Panel of
  Diverse Models (2404.18796)
text
 with the SOA models for a particular language work? i.e., choose 4 of the best open LLMs for Arabic and use those at the pool of raters rather than relying on one powerful judge LLM? 
 with the SOA models for a particular language work? i.e., choose 4 of the best open LLMs for Arabic and use those at the pool of raters rather than relying on one powerful judge LLM? 
As part of the Data is Better Together MPEP project, we are now at the point where some translation efforts have successfully translated 500 highly ranked prompts into a new target language (amazing work from @Rijgersberg et al!)

Our next step is to use these translated prompts to evaluate the performance of LLMs for non English languages. 

Does LLM, as a judge, work outside of English?

Ideally, it would be compelling to leverage LLMs to judge models for non-English since this significantly lowers the barrier to evaluating models (although it doesn't remove this barrier altogether). 

What we want to know is:
- does auto/LLM eval work in general for a particular language
- which model(s) works best as a judge
- do LLMs' judgments of non-English models match human preferences? 

We're starting to think about how to approach this. If you have any ideas of possible approaches feel free to comment or join the discussion here: https://github.com/huggingface/data-is-better-together/issues/61 

Other ideas...

Could an approach like https://huggingface.co/papers/2404.18796 with the SOA models for a particular language work? i.e., choose 4 of the best open LLMs for Arabic and use those at the pool of raters rather than relying on one powerful judge LLM? 
🤗
NePe kristaller486 mmhamdy Rijgersberg alvarobartt
🔥
NePe mmhamdy