text
🇷🇸 New Benchmark for Serbian Language 🇷🇸
🇷🇸 New Benchmark for Serbian Language 🇷🇸
new_line
new_line
mention
@DjMel
DjMel
text
 and I recently released a new benchmark for Serbian language that measures General Knowledge of LLMs. We had to parse over 20 years of university entrance exams for University of Belgrade, so the dataset is of high quality.
 and I recently released a new benchmark for Serbian language that measures General Knowledge of LLMs. We had to parse over 20 years of university entrance exams for University of Belgrade, so the dataset is of high quality.
new_line
new_line
text
🥇 OAI models still hold the podium places with a significant gap compared to open-source models
🥇 OAI models still hold the podium places with a significant gap compared to open-source models
new_line
text
🤔 
🤔 
resource
https://huggingface.co/Qwen/Qwen2-7B-Instruct
model
Qwen/Qwen2-7B-Instruct
https://huggingface.co/Qwen/Qwen2-7B-Instruct
text
 and 
 and 
resource
https://huggingface.co/VAGOsolutions/Llama-3-SauerkrautLM-8b-Instruct
model
VAGOsolutions/Llama-3-SauerkrautLM-8b-Instruct
https://huggingface.co/VAGOsolutions/Llama-3-SauerkrautLM-8b-Instruct
text
 models show promising results considering they weren't trained on Serbian language
 models show promising results considering they weren't trained on Serbian language
new_line
text
📈 Best open-source model seems to be 
📈 Best open-source model seems to be 
resource
https://huggingface.co/Stopwolf/Mustra-7B-Instruct-v0.2
model
Stopwolf/Mustra-7B-Instruct-v0.2
https://huggingface.co/Stopwolf/Mustra-7B-Instruct-v0.2
text
, a merge between 
, a merge between 
resource
https://huggingface.co/gordicaleksa/YugoGPT
model
gordicaleksa/YugoGPT
https://huggingface.co/gordicaleksa/YugoGPT
text
 and 
 and 
resource
https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2
model
mistralai/Mistral-7B-Instruct-v0.2
https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2
new_line
text
📉 Some models like 
📉 Some models like 
resource
https://huggingface.co/google/gemma-2-9b-it
model
google/gemma-2-9b-it
https://huggingface.co/google/gemma-2-9b-it
text
 turned out to be a disappointment with random guessing-like accuracy
 turned out to be a disappointment with random guessing-like accuracy
new_line
new_line
text
Take a look at the whole results at the dataset page:
Take a look at the whole results at the dataset page:
new_line
resource
https://huggingface.co/datasets/DjMel/oz-eval
dataset
DjMel/oz-eval
https://huggingface.co/datasets/DjMel/oz-eval
new_line
new_line
text
P.S. If you have any constructive criticism or ideas for improvement, feel free to use dataset's Discussions page!
P.S. If you have any constructive criticism or ideas for improvement, feel free to use dataset's Discussions page!
🇷🇸 New Benchmark for Serbian Language 🇷🇸

@DjMel and I recently released a new benchmark for Serbian language that measures General Knowledge of LLMs. We had to parse over 20 years of university entrance exams for University of Belgrade, so the dataset is of high quality.

🥇 OAI models still hold the podium places with a significant gap compared to open-source models
🤔 https://huggingface.co/Qwen/Qwen2-7B-Instruct and https://huggingface.co/VAGOsolutions/Llama-3-SauerkrautLM-8b-Instruct models show promising results considering they weren't trained on Serbian language
📈 Best open-source model seems to be https://huggingface.co/Stopwolf/Mustra-7B-Instruct-v0.2, a merge between https://huggingface.co/gordicaleksa/YugoGPT and https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2
📉 Some models like https://huggingface.co/google/gemma-2-9b-it turned out to be a disappointment with random guessing-like accuracy

Take a look at the whole results at the dataset page:
https://huggingface.co/datasets/DjMel/oz-eval

P.S. If you have any constructive criticism or ideas for improvement, feel free to use dataset's Discussions page!
🚀
DjMel Ramikan-BR intellya22
👍
monsoon-nlp Ramikan-BR