German validation dataset from WECHSEL () to evaluate LLM perplexity.

JSON-line files (on JSON object per line):

- `valid.json.gz`: Gzipped validation set as generated by the paper (163,698 docs)
- `valid.random_1636.json.gz`: Random 1% (1636 docs) of the validation set


task_categories:text-generation task_ids:language-modeling task_ids:masked-language-modeling size_categories:100k<n<1M language:de region:us
{"language": ["de"], "size_categories": ["100k<n<1M"], "task_categories": ["text-generation"], "task_ids": ["language-modeling", "masked-language-modeling"]}
TAGS
#task_categories-text-generation #task_ids-language-modeling #task_ids-masked-language-modeling #size_categories-100k<n<1M #language-German #region-us 

German validation dataset from WECHSEL () to evaluate LLM perplexity.

JSON-line files (on JSON object per line):

- 'URL': Gzipped validation set as generated by the paper (163,698 docs)
- 'valid.random_1636.URL': Random 1% (1636 docs) of the validation set


TAGS
#task_categories-text-generation #task_ids-language-modeling #task_ids-masked-language-modeling #size_categories-100k<n<1M #language-German #region-us 
