language:kwn language:nno language:mlg language:miu language:mhi language:yua language:dga language:pol language:sck language:nuj language:ben language:san language:luo language:guz language:hus language:adh language:lwg language:lue language:nhw language:mer language:lug language:xsm language:ell language:rus language:afr language:ewe language:yue language:mnw language:laj language:myx language:fra language:adx language:teo language:cce language:kln language:hat language:zne language:srp language:mmc language:mal language:fat language:nyu language:ndo language:ven language:hch language:ssw language:kqn language:mhw language:koo language:prs language:nso language:yor language:zho language:naq language:nle language:mqu language:lun language:tuv language:ocu language:sme language:kdj language:alz language:lit language:spa language:mfe language:maz language:tum language:nhe language:hun language:dje language:ori language:swa language:ron language:her language:urd language:ttj language:ktz language:tur language:kam language:sag language:kru language:kok language:toi language:jpn language:orm language:rki language:tsn language:nep language:tha language:zul language:ctu language:khg language:dag language:pcm language:keo language:lko language:amh language:saq language:jam language:ara language:kik language:toh language:kan language:lgg language:tam language:aeb language:ckb language:deu language:guj language:ukr language:tir language:tet language:mar language:bxk language:gur language:vie language:old language:nch language:kpz language:xho language:crk language:ita language:kmr language:nyn language:por language:kri language:gaa language:hin language:asm language:mas language:xog language:khm language:csw language:nor language:tgl language:kin language:luc language:ful language:sqi language:kua language:cat language:tsc language:pus language:nld language:kor language:sot language:mya language:lat language:bod language:eng language:nob language:nzi language:twi language:hau language:dan language:kau language:pan language:swe language:fas language:som language:tso language:loz language:anu language:tel language:ada language:nbl language:lsm language:ach language:bem language:pmq language:mat language:gjn language:nya language:epo license:cc0-1.0 arxiv:2309.13320 region:us
---
license: cc0-1.0
language:
- kwn
- nno
- mlg
- miu
- mhi
- yua
- dga
- pol
- sck
- nuj
- ben
- san
- luo
- guz
- hus
- adh
- lwg
- lue
- nhw
- mer
- lug
- xsm
- ell
- rus
- afr
- ewe
- yue
- mnw
- laj
- myx
- fra
- adx
- teo
- cce
- kln
- hat
- zne
- srp
- mmc
- mal
- fat
- nyu
- ndo
- ven
- hch
- ssw
- kqn
- mhw
- koo
- prs
- nso
- yor
- zho
- naq
- nle
- mqu
- lun
- tuv
- ocu
- sme
- kdj
- alz
- lit
- spa
- mfe
- maz
- tum
- nhe
- hun
- dje
- ori
- swa
- ron
- her
- urd
- ttj
- ktz
- tur
- kam
- sag
- kru
- kok
- toi
- jpn
- orm
- rki
- tsn
- nep
- tha
- zul
- ctu
- khg
- dag
- pcm
- keo
- lko
- amh
- saq
- jam
- ara
- kik
- toh
- kan
- lgg
- tam
- aeb
- ckb
- deu
- guj
- ukr
- tir
- tet
- mar
- bxk
- gur
- vie
- old
- nch
- kpz
- xho
- crk
- ita
- kmr
- nyn
- por
- kri
- gaa
- hin
- asm
- mas
- xog
- khm
- csw
- nor
- tgl
- kin
- luc
- ful
- sqi
- kua
- cat
- tsc
- pus
- nld
- kor
- sot
- mya
- lat
- bod
- eng
- nob
- nzi
- twi
- hau
- dan
- kau
- pan
- swe
- fas
- som
- tso
- loz
- anu
- tel
- ada
- nbl
- lsm
- ach
- bem
- pmq
- mat
- gjn
- nya
- epo

pretty_name: GlotStoryBook Corpus
---

## Dataset Description

Story Books for 180 ISO-639-3 script pairs (174 unique ISO-639-3 codes).

- **Homepage:** [homepage](https://github.com/cisnlp/GlotStoryBook)
- **Repository:** [github](https://github.com/cisnlp/GlotStoryBook)
- **Paper:** [paper]()
- **Point of Contact:** amir@cis.lmu.de


## Usage (HF Loader)
```python
from datasets import load_dataset
dataset = load_dataset('cis-lmu/GlotStoryBook')
print(dataset['train'][0]) # First row data
```

## Download
If you are not a fan of the HF dataloader, download it directly:

```python
! wget https://huggingface.co/datasets/cis-lmu/GlotStoryBook/resolve/main/GlotStoryBook.csv
```

# Tools

To compute the script of each text we used Glotscript ([code](https://github.com/cisnlp/GlotScript) and [paper](https://arxiv.org/abs/2309.13320)).


## License and Copyright
We do not own any of the text from which these data has been extracted.
All the files are collected from the repository located at https://github.com/global-asp/.
The source repository for each text and file is stored in the dataset.
Each file in the dataset is associated with one license from the CC family.
The licenses include 'CC BY', 'CC BY-NC', 'CC BY-NC-SA', 'CC-BY', 'CC-BY-NC', and 'Public Domain'.
We also license the code, actual packaging and the metadata of these data under the cc0-1.0.

## Github
We additionally provide a GitHub version that openly shares the source code for processing this dataset:
https://github.com/cisnlp/GlotStoryBook


## Citation
If you use any part of this code and data in your research, please cite it (along with https://github.com/global-asp/) using the following BibTeX entry.
This work is part of the [GlotLID](https://github.com/cisnlp/GlotLID) project.

```
@inproceedings{
  kargaran2023glotlid,
  title={{GlotLID}: Language Identification for Low-Resource Languages},
  author={Kargaran, Amir Hossein and Imani, Ayyoob and Yvon, Fran{\c{c}}ois and Sch{\"u}tze, Hinrich},
  booktitle={The 2023 Conference on Empirical Methods in Natural Language Processing},
  year={2023},
  url={https://openreview.net/forum?id=dl4e3EBz5j}
}

```