multilinguality:multilingual language:tir language:rmn language:arb language:mxv language:mal language:fij language:som language:cot language:fur language:vie language:zlm language:bam language:chr language:maz language:yad language:ztu language:ykg language:ccp language:alt language:ayr language:njo language:bci language:gyr language:run language:haw language:rgn language:cak language:kwi language:fra language:agr language:duu language:ilo language:nhn language:kdh language:cnh language:bod language:mya language:ady language:pol language:ydd language:cos language:lot language:arl language:glv language:gag language:bfa language:afr language:lij language:ibb language:toi language:tzm language:ron language:ojb language:san language:eng language:bum language:pam language:kqs language:dje language:auc language:smo language:por language:fry language:lad language:pov language:tyv language:guc language:huu language:ese language:kbp language:eve language:yrk language:lin language:tdt language:qvc language:top language:nav language:twi language:oss language:lia language:ame language:hun language:lit language:que language:qug language:nku language:csa language:lao language:knc language:kjh language:jav language:mam language:ita language:ppl language:aar language:tbz language:ssw language:bug language:srp language:kaz language:min language:mad language:orh language:tgk language:kat language:uig language:tzo language:hat language:shn language:kbd language:niv language:idu language:krl language:abk language:mto language:gla language:ijs language:cri language:uzn language:tah language:tob language:kir language:quy language:hnj language:srr language:lvs language:nan language:hns language:snk language:swh language:ekk language:guu language:div language:dzo language:spa language:hms language:ell language:ace language:war language:ind language:cjy language:cfm language:nds language:ewe language:tel language:src language:fuf language:vmw language:zro language:men language:kqn language:nzi language:taj language:khk language:ddn language:nso language:mxi language:pon language:fvr language:hau language:ktu language:tem language:yor language:pnb language:ltz language:evn language:cjs language:nba language:niu language:dan language:acu language:zgh language:chj language:heb language:lua language:quz language:cbi language:cpu language:wuu language:mah language:kmb language:mcd language:ben language:lus language:ajg language:azj language:tha language:dga language:isl language:sus language:fkv language:jiv language:mor language:nio language:als language:buc language:kde language:nbl language:ceb language:ven language:sun language:cbt language:swb language:tur language:dyo language:sin language:pbu language:ada language:pap language:qvh language:loz language:pan language:qva language:sme language:bax language:tuk language:hsb language:hus language:qvn language:ban language:cha language:zyb language:hin language:tat language:qxu language:gej language:quc language:mnw language:bho language:udu language:kha language:kbr language:tsz language:pau language:mkd language:shp language:ike language:lue language:tgl language:yap language:yua language:koi language:hrv language:emk language:tet language:ndo language:cbu language:vep language:cmn language:sag language:nym language:rus language:gjn language:guk language:kri language:ote language:lun language:vai language:bis language:arn language:tsn language:gle language:hak language:gkp language:ura language:tca language:xho language:wln language:amc language:mos language:lld language:bul language:qxn language:bcl language:ctd language:dip language:dag language:kek language:bre language:mri language:fin language:sah language:cym language:kan language:fao language:gsw language:sey language:bem language:bos language:bin language:chv language:tpi language:ami language:oaa language:lob language:ast language:nno language:sco language:khm language:pes language:pbb language:tam language:ibo language:sid language:plt language:guj language:hsn language:kin language:lug language:slr language:koo language:xsm language:jpn language:oki language:deu language:rar language:pcm language:hni language:vec language:gld language:sot language:crs language:fuv language:npi language:nya language:kea language:blt language:roh language:cbr language:chk language:kal language:mfq language:quh language:kor language:slv language:cof language:shk language:zul language:qwh language:fon language:mic language:prs language:mag language:bel language:iii language:mar language:dyu language:boa language:swe language:pis language:mlt language:amh language:umb language:cnr language:mai language:toj language:csw language:ina language:bba language:cbs language:kng language:oci language:pcd language:miq language:lat language:qvm language:wwa language:urd language:kmr language:ido language:gaa language:epo language:gaz language:cat language:hye language:cni language:suk language:gug language:gan language:cjk language:tzh language:zam language:ces language:cic language:mcf language:not language:kaa language:tso language:piu language:fat language:mzi language:snn language:tly language:eus language:nld language:nob language:wol language:hlt language:sna language:tiv language:ton language:hea language:skr language:lns language:rup language:cab language:glg language:yao language:nyn language:aii language:slk language:ukr language:kkh language:zdj language:amr language:yue language:crh language:hil license:cc0-1.0 UDHR udhr language identification arxiv:2309.13320 region:us
    UDHR-LID Corpus \
---
license: cc0-1.0
multilinguality:
  - multilingual
language:
- tir
- rmn
- arb
- mxv
- mal
- fij
- som
- cot
- fur
- vie
- zlm
- bam
- chr
- maz
- yad
- ztu
- ykg
- ccp
- alt
- ayr
- njo
- bci
- gyr
- run
- haw
- rgn
- cak
- kwi
- fra
- agr
- duu
- ilo
- nhn
- kdh
- cnh
- bod
- mya
- ady
- pol
- ydd
- cos
- lot
- arl
- glv
- gag
- bfa
- afr
- lij
- zlm
- ibb
- toi
- tzm
- ron
- ojb
- san
- eng
- bum
- pam
- kqs
- dje
- auc
- smo
- por
- fry
- lad
- pov
- tyv
- guc
- huu
- ese
- kbp
- eve
- yrk
- lin
- tdt
- qvc
- top
- nav
- twi
- oss
- lia
- ame
- hun
- lit
- que
- qug
- nku
- csa
- lao
- knc
- kjh
- jav
- mam
- ita
- ppl
- aar
- tbz
- ssw
- bug
- srp
- kaz
- min
- mad
- orh
- tgk
- kat
- uig
- tzo
- hat
- shn
- kbd
- niv
- idu
- krl
- abk
- mto
- gla
- ijs
- cri
- uzn
- tah
- tob
- kir
- quy
- hnj
- srr
- lvs
- nan
- hns
- snk
- swh
- ekk
- guu
- div
- dzo
- spa
- hms
- ell
- ace
- war
- ind
- cjy
- cfm
- nds
- ewe
- tel
- src
- fuf
- vmw
- zro
- men
- kqn
- nzi
- taj
- khk
- ddn
- nso
- mxi
- pon
- fvr
- hau
- ktu
- tem
- yor
- pnb
- ltz
- evn
- cjs
- nba
- niu
- dan
- acu
- zgh
- chj
- heb
- lua
- quz
- uig
- cbi
- jav
- cpu
- wuu
- mah
- kmb
- mcd
- ben
- lus
- ajg
- azj
- tha
- dga
- isl
- sus
- fuf
- fkv
- jiv
- mor
- nio
- als
- buc
- kde
- nbl
- ceb
- ven
- sun
- cbt
- swb
- tur
- dyo
- sin
- pbu
- ada
- pap
- qvh
- loz
- pan
- qva
- sme
- bax
- tuk
- hsb
- hus
- qvn
- ban
- cha
- zyb
- hin
- tat
- uzn
- qxu
- gej
- quc
- mnw
- bho
- udu
- kha
- kbr
- tsz
- pau
- mkd
- shp
- ike
- lue
- tgl
- yap
- yua
- koi
- hrv
- emk
- tet
- ndo
- cbu
- vep
- cmn
- sag
- nym
- rus
- gjn
- guk
- kri
- ote
- lun
- vai
- bis
- arn
- tsn
- gle
- hak
- gkp
- ura
- tca
- xho
- wln
- amc
- mos
- lld
- bul
- qxn
- bcl
- ctd
- dip
- dag
- kek
- bre
- mri
- fin
- sah
- cym
- kan
- fao
- gsw
- sey
- bem
- bos
- bin
- chv
- tpi
- ami
- oaa
- lob
- ast
- nno
- sco
- tuk
- khm
- pes
- pbb
- tam
- ibo
- san
- sid
- plt
- guj
- hsn
- kin
- lug
- slr
- koo
- xsm
- jpn
- oki
- deu
- rar
- pcm
- hni
- vec
- gld
- sot
- crs
- fuv
- srp
- npi
- nya
- kea
- blt
- roh
- cbr
- chk
- kal
- mfq
- quh
- kor
- slv
- cof
- shk
- zul
- qwh
- fon
- mic
- prs
- mag
- bel
- iii
- mar
- dyu
- boa
- swe
- pis
- mlt
- amh
- umb
- cnr
- mai
- toj
- csw
- ina
- bba
- cbs
- kng
- oci
- pcd
- miq
- lat
- qvm
- wwa
- bos
- urd
- kmr
- ido
- gaa
- epo
- gaz
- cat
- hye
- cni
- suk
- gug
- gan
- cjk
- tzh
- zam
- ces
- cic
- mcf
- not
- kaa
- tso
- piu
- fat
- mzi
- snn
- tly
- eus
- nld
- nob
- wol
- hlt
- sna
- tiv
- ton
- hea
- skr
- lns
- rup
- cab
- glg
- tgl
- yao
- nyn
- aii
- tzm
- slk
- ukr
- kkh
- zdj
- amr
- yue
- crh
- hil
tags:
- UDHR
- udhr
- language identification
pretty_name: UDHR-LID
---

# UDHR-LID

**Why UDHR-LID?**

You can access UDHR [here](http://www.unicode.org/udhr/d/), but when a verse is missing, they have texts such as "missing" or "?". Also, about 1/3 of the sentences consist only of "articles 1-30" in different languages. We cleaned the entire dataset from XML files and selected only the paragraphs. We cleared any unrelated language texts from the data and also removed the cases that were incorrect.

Incorrect? Look at the ckb and kmr files in the UDHR. Both are the same! ckb is known for the Arabic script, although it can also be written in Latin. Clearly, a unique file cannot belong to two different languages. We also deleted files that we believe those scripts are no longer in use.

The deleted files include:
- ckb_Latn (Arabic is in use.)
- azb_Latn (Arabic is in use.)
- azj_Cyrl (Latin is in use.)
- khk_Mong (Cyrillic is in use.)
- vie_Hani (Latin is in use.)

For dealing with scripts in other languages, if you are interested, check Glotscript [code](https://github.com/cisnlp/GlotScript) and [paper](https://arxiv.org/abs/2309.13320). We have prepared a tool for detecting the script of a text, as well as metadata to determine the correct script for each language.

We believe UDHR should remain a test corpus in NLP, not a training corpus. Of course, we are not opposed to great works such as Franc built on top of UDHR. However, if your work scale iis much bigger than UDHR, do not put UDHR in your data. Use it as test/validation, or find out what is wrong with your training data. Be aware that a part of UDHR may be hosted on other websites such as Wikipedia or news websites like BBC, or collaborative translation communities like Tatoeba. Before using UDHR as a test, exclude any sentence where UDHR is a substring of it in your training.
We created this corpus for language identification evaluation task, but feel free to use it for your own task. The texts here are not in order, and they're not parallel. However, each row of data belongs to the determined language, cleaned, and has rich linguistic content! 

## Usage (HF Loader)

Substitute `aar_Latn` with your preferred language:

```python
from datasets import load_dataset
dataset = load_dataset('cis-lmu/UDHR-LID', 'aar_Latn', split='test')
print(dataset[0]) # First row data of aar_Latn
```

## Download
If you are not a fan of the HF dataloader, download each language directly:

```python
! wget https://huggingface.co/datasets/cis-lmu/UDHR-LID/resolve/main/aar_Latn.txt 
```


## License
UDHR is the most translated copyright-free document in the world.
We license the actual packaging, the metadata and the annotations of these data under the cc0-1.0 (waiving all of the rights under copyright law).


## Citation
If you use any part of this data in your research , please cite it (along with http://www.unicode.org/udhr/d/) using the following BibTeX entry.

```
@inproceedings{
  kargaran2023glotlid,
  title={{GlotLID}: Language Identification for Low-Resource Languages},
  author={Kargaran, Amir Hossein and Imani, Ayyoob and Yvon, Fran{\c{c}}ois and Sch{\"u}tze, Hinrich},
  booktitle={The 2023 Conference on Empirical Methods in Natural Language Processing},
  year={2023},
  url={https://openreview.net/forum?id=dl4e3EBz5j}
}
```
