# Massive Multitask Language Understanding (MMLU) in German

This dataset is to be used for the evaluation of LLM German language understanding. 
It is based on the hendrycksTest dataset ([here](https://huggingface.co/datasets/cais/mmlu) and [here](https://huggingface.co/datasets/tasksource/mmlu)) and was created
by using the GPT-3.5 API to translate the entire test set and a few examples of the validation set. To make sure the answer options follow the intended sentence structure
and are always of the correct format, GPT was prompted to output in a JSON format. This came with some complications that were later manually fixed.

The prompt used to translate a single example was the following:
```
insert prompt here @TODO
```

This translation cost a total of ~13€ including iterating on the prompt and fixing broken examples.


license:mit region:us
{"license": "mit"}
TAGS
#license-mit #region-us 

# Massive Multitask Language Understanding (MMLU) in German

This dataset is to be used for the evaluation of LLM German language understanding. 
It is based on the hendrycksTest dataset (here and here) and was created
by using the GPT-3.5 API to translate the entire test set and a few examples of the validation set. To make sure the answer options follow the intended sentence structure
and are always of the correct format, GPT was prompted to output in a JSON format. This came with some complications that were later manually fixed.

The prompt used to translate a single example was the following:


This translation cost a total of ~13€ including iterating on the prompt and fixing broken examples.


# Massive Multitask Language Understanding (MMLU) in German

This dataset is to be used for the evaluation of LLM German language understanding. 
It is based on the hendrycksTest dataset (here and here) and was created
by using the GPT-3.5 API to translate the entire test set and a few examples of the validation set. To make sure the answer options follow the intended sentence structure
and are always of the correct format, GPT was prompted to output in a JSON format. This came with some complications that were later manually fixed.

The prompt used to translate a single example was the following:


This translation cost a total of ~13€ including iterating on the prompt and fixing broken examples.
TAGS
#license-mit #region-us 
 # Massive Multitask Language Understanding (MMLU) in German

This dataset is to be used for the evaluation of LLM German language understanding. 
It is based on the hendrycksTest dataset (here and here) and was created
by using the GPT-3.5 API to translate the entire test set and a few examples of the validation set. To make sure the answer options follow the intended sentence structure
and are always of the correct format, GPT was prompted to output in a JSON format. This came with some complications that were later manually fixed.

The prompt used to translate a single example was the following:


This translation cost a total of ~13€ including iterating on the prompt and fixing broken examples.