
## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Languages](#languages)

- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
<!-- - [Dataset Creation](#dataset-creation) 
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions) -->

## Dataset Description

- **Homepage:** [SIL AI](https://ai.sil.org/)
- **Point of Contact:** [SIL AI email](mailto:idx_aqua@sil.org)
- **Source Data:** [Bloom Library](https://bloomlibrary.org/)

![logo for Bloom Library](https://bloom-vist.s3.amazonaws.com/bloom_logo.png) ![sil-ai logo](https://s3.amazonaws.com/moonup/production/uploads/1661440873726-6108057a823007eaf0c7bd10.png) 

## Dataset Summary

**Bloom** is free, open-source software and an associated website [Bloom Library](https://bloomlibrary.org/), app, and services developed by [SIL International](https://www.sil.org/). Bloom’s primary goal is to equip non-dominant language communities and their members to create the literature they want for their community and children. Bloom also serves organizations that help such communities develop literature and education or other aspects of community development.

This version of the Bloom Library data is developed specifically for the language modeling task. It includes data from 364 languages across 31 language families. There is a mean of 32 stories and median of 2 stories per language. 

**Note**: If you speak one of these languages and can help provide feedback or corrections, please let us know!

**Note**: Although this data was used in the training of the [BLOOM model](https://huggingface.co/bigscience/bloom), this dataset only represents a small portion of the data used to train that model. Data from "Bloom Library" was combined with a large number of other datasets to train that model. "Bloom Library" is a project that existed prior to the BLOOM model, and is something separate. All that to say... We were using the "Bloom" name before it was cool. 😉  

## Languages

Of the 500+ languages listed at BloomLibrary.org, there are 363 languages available in this dataset. Here are the corresponding ISO 639-3 codes:

aaa, abc, ada, adq, aeu, afr, agq, ags, ahk, aia, ajz, aka, ame, amh, amp, amu, ann, aph, awa, awb, azn, azo, bag, bam, baw, bax, bbk, bcc, bce, bec, bef, ben, bfd, bfm, bfn, bgf, bho, bhs, bis, bjn, bjr, bkc, bkh, bkm, bkx, bob, bod, boz, bqm, bra, brb, bri, brv, bss, bud, buo, bwt, bwx, bxa, bya, bze, bzi, cak, cbr, ceb, cgc, chd, chp, cim, clo, cmn, cmo, csw, cuh, cuv, dag, ddg, ded, deu, dig, dje, dmg, dnw, dtp, dtr, dty, dug, eee, ekm, enb, enc, eng, ewo, fas, fil, fli, fon, fra, fub, fuh, gal, gbj, gou, gsw, guc, guj, guz, gwc, hao, hat, hau, hbb, hig, hil, hin, hla, hna, hre, hro, idt, ilo, ind, ino, isu, ita, jgo, jmx, jpn, jra, kak, kam, kan, kau, kbq, kbx, kby, kek, ken, khb, khm, kik, kin, kir, kjb, kmg, kmr, kms, kmu, kor, kqr, krr, ksw, kur, kvt, kwd, kwu, kwx, kxp, kyq, laj, lan, lao, lbr, lfa, lgg, lgr, lhm, lhu, lkb, llg, lmp, lns, loh, lsi, lts, lug, luy, lwl, mai, mal, mam, mar, mdr, mfh, mfj, mgg, mgm, mgo, mgq, mhx, miy, mkz, mle, mlk, mlw, mmu, mne, mnf, mnw, mot, mqj, mrn, mry, msb, muv, mve, mxu, mya, myk, myx, mzm, nas, nco, nep, new, nge, ngn, nhx, njy, nla, nld, nlv, nod, nsk, nsn, nso, nst, nuj, nwe, nwi, nxa, nxl, nya, nyo, nyu, nza, odk, oji, oki, omw, ori, ozm, pae, pag, pan, pbt, pce, pcg, pdu, pea, pex, pis, pkb, pmf, pnz, por, psp, pwg, qub, quc, quf, quz, qve, qvh, qvm, qvo, qxh, rel, rnl, ron, roo, rue, rug, rus, san, saq, sat, sdk, sea, sgd, shn, sml, snk, snl, som, sot, sox, spa, sps, ssn, stk, swa, swh, sxb, syw, taj, tam, tbj, tdb, tdg, tdt, teo, tet, tgk, tha, the, thk, thl, thy, tio, tkd, tnl, tnn, tnp, tnt, tod, tom, tpi, tpl, tpu, tsb, tsn, tso, tuv, tuz, tvs, udg, unr, urd, uzb, ven, vie, vif, war, wbm, wbr, wms, wni, wnk, wtk, xho, xkg, xmd, xmg, xmm, xog, xty, yas, yav, ybb, ybh, ybi, ydd, yea, yet, yid, yin, ymp, zaw, zho, zlm, zuh, zul

## Dataset Statistics
Some of the languages included in the dataset just include 1 or a couple of "stories." These are not split between training, validation, and test. For those with higher numbers of available stories we include the following numbers of stories in each split:

| ISO 639-3   | Name                          |   Train Stories |   Validation Stories |   Test Stories |
|:------------|:------------------------------|----------------:|---------------------:|---------------:|
| aeu         | Akeu                          |              47 |                    6 |              5 |
| afr         | Afrikaans                     |              19 |                    2 |              2 |
| ahk         | Akha                          |              81 |                   10 |             10 |
| aph         | Athpariya                     |              28 |                    4 |              3 |
| awa         | Awadhi                        |             131 |                   16 |             16 |
| ben         | Bengali                       |             201 |                   25 |             25 |
| bfn         | Bunak                         |              11 |                    1 |              1 |
| bho         | Bhojpuri                      |             139 |                   17 |             17 |
| bis         | Bislama                       |              20 |                    2 |              2 |
| bkm         | Kom (Cameroon)                |              15 |                    2 |              1 |
| bkx         | Baikeno                       |               8 |                    1 |              1 |
| brb         | Brao                          |              18 |                    2 |              2 |
| bwx         | Bu-Nao Bunu                   |              14 |                    2 |              1 |
| bzi         | Bisu                          |              53 |                    7 |              6 |
| cak         | Kaqchikel                     |              54 |                    7 |              6 |
| cbr         | Cashibo-Cacataibo             |              11 |                    1 |              1 |
| ceb         | Cebuano                       |             335 |                   42 |             41 |
| cgc         | Kagayanen                     |             158 |                   20 |             19 |
| cmo         | Central Mnong                 |              16 |                    2 |              2 |
| ddg         | Fataluku                      |              14 |                    2 |              1 |
| deu         | German                        |              36 |                    4 |              4 |
| dtp         | Kadazan Dusun                 |              13 |                    2 |              1 |
| dty         | Dotyali                       |             138 |                   17 |             17 |
| eng         | English                       |            2107 |                  263 |            263 |
| fas         | Persian                       |             104 |                   13 |             12 |
| fil         | Filipino                      |              55 |                    7 |              6 |
| fra         | French                        |             323 |                   40 |             40 |
| gal         | Galolen                       |              11 |                    1 |              1 |
| gwc         | Gawri                         |              15 |                    2 |              1 |
| hat         | Haitian                       |             208 |                   26 |             26 |
| hau         | Hausa                         |             205 |                   26 |             25 |
| hbb         | Huba                          |              22 |                    3 |              2 |
| hin         | Hindi                         |              16 |                    2 |              2 |
| idt         | Idaté                         |               8 |                    1 |              1 |
| ind         | Indonesian                    |             208 |                   26 |             25 |
| jmx         | Western Juxtlahuaca Mixtec    |              19 |                    2 |              2 |
| jra         | Jarai                         |             112 |                   14 |             13 |
| kak         | Kalanguya                     |             156 |                   20 |             19 |
| kan         | Kannada                       |              17 |                    2 |              2 |
| kau         | Kanuri                        |              36 |                    5 |              4 |
| kek         | Kekchí                        |              29 |                    4 |              3 |
| khb         | Lü                            |              25 |                    3 |              3 |
| khm         | Khmer                         |              28 |                    4 |              3 |
| kik         | Kikuyu                        |               8 |                    1 |              1 |
| kir         | Kirghiz                       |             306 |                   38 |             38 |
| kjb         | Q'anjob'al                    |              82 |                   10 |             10 |
| kmg         | Kâte                          |              16 |                    2 |              1 |
| kor         | Korean                        |             106 |                   13 |             13 |
| krr         | Krung                         |              24 |                    3 |              3 |
| kwd         | Kwaio                         |              19 |                    2 |              2 |
| kwu         | Kwakum                        |              16 |                    2 |              2 |
| lbr         | Lohorung                      |               8 |                    1 |              1 |
| lhu         | Lahu                          |              32 |                    4 |              4 |
| lsi         | Lashi                         |              21 |                    3 |              2 |
| mai         | Maithili                      |             144 |                   18 |             18 |
| mal         | Malayalam                     |              12 |                    1 |              1 |
| mam         | Mam                           |             108 |                   13 |             13 |
| mar         | Marathi                       |               8 |                    1 |              1 |
| mgm         | Mambae                        |              12 |                    2 |              1 |
| mhx         | Maru                          |              79 |                   10 |              9 |
| mkz         | Makasae                       |              16 |                    2 |              2 |
| mya         | Burmese                       |              31 |                    4 |              3 |
| myk         | Mamara Senoufo                |              28 |                    3 |              3 |
| nep         | Nepali (macrolanguage)        |             160 |                   20 |             20 |
| new         | Newari                        |             142 |                   18 |             17 |
| nlv         | Orizaba Nahuatl               |               8 |                    1 |              1 |
| nsn         | Nehan                         |               9 |                    1 |              1 |
| nwi         | Southwest Tanna               |               9 |                    1 |              1 |
| nxa         | Nauete                        |              12 |                    1 |              1 |
| omw         | South Tairora                 |              10 |                    1 |              1 |
| pbt         | Southern Pashto               |             164 |                   21 |             20 |
| pce         | Ruching Palaung               |              30 |                    4 |              3 |
| pis         | Pijin                         |              14 |                    2 |              1 |
| por         | Portuguese                    |             131 |                   16 |             16 |
| quc         | K'iche'                       |              80 |                   10 |              9 |
| rus         | Russian                       |             283 |                   35 |             35 |
| sdk         | Sos Kundi                     |               9 |                    1 |              1 |
| snk         | Soninke                       |              28 |                    4 |              3 |
| spa         | Spanish                       |             423 |                   53 |             52 |
| swh         | Swahili (individual language) |              58 |                    7 |              7 |
| tam         | Tamil                         |              13 |                    2 |              1 |
| tdg         | Western Tamang                |              26 |                    3 |              3 |
| tdt         | Tetun Dili                    |              22 |                    3 |              2 |
| tet         | Tetum                         |               8 |                    1 |              1 |
| tgk         | Tajik                         |              24 |                    3 |              2 |
| tha         | Thai                          |             228 |                   29 |             28 |
| the         | Chitwania Tharu               |              11 |                    1 |              1 |
| thl         | Dangaura Tharu                |             148 |                   19 |             18 |
| tnl         | Lenakel                       |              10 |                    1 |              1 |
| tnn         | North Tanna                   |               9 |                    1 |              1 |
| tpi         | Tok Pisin                     |             161 |                   20 |             20 |
| tpu         | Tampuan                       |              24 |                    3 |              2 |
| uzb         | Uzbek                         |              24 |                    3 |              2 |
| war         | Waray (Philippines)           |              16 |                    2 |              2 |
| wbr         | Wagdi                         |              10 |                    1 |              1 |
| wni         | Ndzwani Comorian              |              12 |                    2 |              1 |
| xkg         | Kagoro                        |              16 |                    2 |              1 |
| ybh         | Yakha                         |              16 |                    2 |              1 |
| zho         | Chinese                       |              34 |                    4 |              4 |
| zlm         | Malay (individual language)   |               8 |                    1 |              1 |
| zul         | Zulu                          |              19 |                    2 |              2 |

## Dataset Structure

### Data Instances

The examples look like this for Hindi:

```
from datasets import load_dataset

# Specify the language code.
dataset = load_dataset("sil-ai/bloom-lm", 'hin')

# A data point consists of stories in the specified language code.
# To see a story:
print(dataset['train']['text'][0])
```

This would produce an output:

```
साबू ने एक कंकड़ को ठोकर मारी। कंकड़ लुढ़कता हुआ एक पेड़ के पास पहुँचा। पेड़ के तने पर मुलायम बाल थे। साबू ने छुए और ऊपर देखा, ऊपर, ऊपर और उससे भी ऊपर...दो आँखें नीचे देख रही थीं।
“हेलो, तुम कौन हो?” साबू को बड़ा अचम्भा हुआ।“हेलो, मैं जिराफ़ हूँ। मेरा नाम है जोजो।  मैं तुम्हारे साथ खेल सकता हूँ। मेरी पीठ पर चढ़ जाओ, मैं तुम्हें घुमा के लाता हूँ।”
साबू जोजो की पीठ पर चढ़ गया और वे सड़क पर चल निकले। फिर पहाड़ी पर और शहर के बीचों बीच।
साबू खुशी से चिल्लाया, “जोजो दाएँ मुड़ो,
                                बाएँ मुड़ो और फिर दाएँ।” अब वे उसकी दोस्त मुन्नी के घर पहुँच गये।
आज मुन्नी का जन्मदिन था। साबू को जोजो पर सवारी करते देख बच्चों ने ताली बजायी। 
                                जोजो ने गुब्बारे लटकाने में आन्टी की मदद करी क्योंकि वह इतना... लम्बा था। 
                                कितना आसान था!
जोजो ने सब बच्चों को सवारी कराई।
                                उनके साथ बॉल भी खेली। बड़े मज़े की पार्टी थी।सब ने गाया, “हैप्पी बर्थ डे टु यू ।”
                                        आन्टी ने मेज़ पर समोसे, गुलाब जामुन और आइसक्रीम सजाई।
जोजो को आइसक्रीम बहुत पसन्द आई। अंकल उसके लिये एक बाल्टी भर के आइसक्रीम लाये। जोजो ने पूरी बाल्टी ख़त्म कर दी।  अब घर जाने का समय हो गया।

सब ने कहा, “बाय बाय जोजो, बाय बाय साबू।” साबू और जोजो घर लौटे।
```

Whereas if you wish to gather all the text for a language you may use this:
```
dataset['train']['text']
```

### Data Fields

The metadata fields below are available and the full dataset will be updated with per story metadata soon (in August 2022). As of now a majority of stories have metadata, but some are missing certain fields. In terms of licenses, all stories included in the current release are released under a Creative Commons license (even if the individual story metadata fields are missing). 

- **text**: the text of the story/book, concatenated together from the different pages. 
- **id**: id of the sample
- **title**: title of the book, e.g. "Going to Buy a Book". 
- **license**: specific license used, e.g. "cc-by-sa" for "Creative Commons, by attribution, share-alike". 
- **copyright**: copyright notice from the original book on bloomlibrary.org
- **pageCount**: page count from the metadata on the original book on bloomlibrary.org. 
- **bookInstanceId**: unique ID for each book/translation assigned by Bloom. For example the Hindi version of 'Going to Buy a Book' is 'af86eefd-f69c-4e06-b8eb-e0451853aab9'.
- **bookLineage**: Unique bookInstanceIDs of _other_ Bloom books that this book is in some way based on. For example, the Hindi version in the example above is based on '056B6F11-4A6C-4942-B2BC-8861E62B03B3'. It's quite possible for this to be either empty, or have multiple entries. For example, the book 'Saboo y Jojo' with ID '5b232a5f-561d-4514-afe7-d6ed2f6a940f' is based on two others, ['056B6F11-4A6C-4942-B2BC-8861E62B03B3', '10a6075b-3c4f-40e4-94f3-593497f2793a']
- (coming soon) **contentLanguages**: Other languages this book may be available in. "Going to Buy a Book" is available in ['eng', 'kan', 'mar', 'pan', 'ben', 'guj', 'hin'] for example.


### Data Splits

All languages include a train, validation, and test split. However, for language having a small number of stories, certain of these splits maybe empty. In such cases, we recommend using any data for testing only or for zero-shot experiments. 

## Changelog
- **25 August 2022** - add the remaining metadata, change data type of `pageCount` to int32
- **24 August 2022** - majority of metadata added back in to the filtered/ clean data
- **23 August 2022** - metadata temporarily removed to update to cleaner dataset
task_ids:language-modeling annotations_creators:expert-generated language_creators:expert-generated multilinguality:multilingual size_categories:10K<n<100K source_datasets:original language:afr language:af language:aaa language:abc language:ada language:adq language:aeu language:agq language:ags language:ahk language:aia language:ajz language:aka language:ak language:ame language:amh language:am language:amp language:amu language:ann language:aph language:awa language:awb language:azn language:azo language:bag language:bam language:bm language:baw language:bax language:bbk language:bcc language:bce language:bec language:bef language:ben language:bn language:bfd language:bfm language:bfn language:bgf language:bho language:bhs language:bis language:bi language:bjn language:bjr language:bkc language:bkh language:bkm language:bkx language:bob language:bod language:bo language:boz language:bqm language:bra language:brb language:bri language:brv language:bss language:bud language:buo language:bwt language:bwx language:bxa language:bya language:bze language:bzi language:cak language:cbr language:ceb language:cgc language:chd language:chp language:cim language:clo language:cmn language:zh language:cmo language:csw language:cuh language:cuv language:dag language:ddg language:ded language:deu language:de language:dig language:dje language:dmg language:dnw language:dtp language:dtr language:dty language:dug language:eee language:ekm language:enb language:enc language:eng language:en language:ewo language:fas language:fa language:fil language:fli language:fon language:fra language:fr language:fub language:fuh language:gal language:gbj language:gou language:gsw language:guc language:guj language:gu language:guz language:gwc language:hao language:hat language:ht language:hau language:ha language:hbb language:hig language:hil language:hin language:hi language:hla language:hna language:hre language:hro language:idt language:ilo language:ind language:id language:ino language:isu language:ita language:it language:jgo language:jmx language:jpn language:ja language:jra language:kak language:kam language:kan language:kn language:kau language:kr language:kbq language:kbx language:kby language:kek language:ken language:khb language:khm language:km language:kik language:ki language:kin language:rw language:kir language:ky language:kjb language:kmg language:kmr language:ku language:kms language:kmu language:kor language:ko language:kqr language:krr language:ksw language:kur language:kvt language:kwd language:kwu language:kwx language:kxp language:kyq language:laj language:lan language:lao language:lo language:lbr language:lfa language:lgg language:lgr language:lhm language:lhu language:lkb language:llg language:lmp language:lns language:loh language:lsi language:lts language:lug language:lg language:luy language:lwl language:mai language:mal language:ml language:mam language:mar language:mr language:mdr language:mfh language:mfj language:mgg language:mgm language:mgo language:mgq language:mhx language:miy language:mkz language:mle language:mlk language:mlw language:mmu language:mne language:mnf language:mnw language:mot language:mqj language:mrn language:mry language:msb language:muv language:mve language:mxu language:mya language:my language:myk language:myx language:mzm language:nas language:nco language:nep language:ne language:new language:nge language:ngn language:nhx language:njy language:nla language:nld language:nl language:nlv language:nod language:nsk language:nsn language:nso language:nst language:nuj language:nwe language:nwi language:nxa language:nxl language:nya language:ny language:nyo language:nyu language:nza language:odk language:oji language:oj language:oki language:omw language:ori language:or language:ozm language:pae language:pag language:pan language:pa language:pbt language:pce language:pcg language:pdu language:pea language:pex language:pis language:pkb language:pmf language:pnz language:por language:pt language:psp language:pwg language:qaa language:qub language:quc language:quf language:quz language:qve language:qvh language:qvm language:qvo language:qxh language:rel language:rnl language:ron language:ro language:roo language:rue language:rug language:rus language:ru language:san language:sa language:saq language:sat language:sdk language:sea language:sgd language:shn language:sml language:snk language:snl language:som language:so language:sot language:st language:sox language:spa language:es language:sps language:ssn language:stk language:swa language:sw language:swh language:sxb language:syw language:taj language:tam language:ta language:tbj language:tdb language:tdg language:tdt language:teo language:tet language:tgk language:tg language:tha language:th language:the language:thk language:thl language:thy language:tio language:tkd language:tnl language:tnn language:tnp language:tnt language:tod language:tom language:tpi language:tpl language:tpu language:tsb language:tsn language:tn language:tso language:ts language:tuv language:tuz language:tvs language:udg language:unr language:urd language:ur language:uzb language:uz language:ven language:ve language:vie language:vi language:vif language:war language:wbm language:wbr language:wms language:wni language:wnk language:wtk language:xho language:xh language:xkg language:xmd language:xmg language:xmm language:xog language:xty language:yas language:yav language:ybb language:ybh language:ybi language:ydd language:yea language:yet language:yid language:yi language:yin language:ymp language:zaw language:zho language:zlm language:zuh language:zul language:zu license:cc-by-4.0 license:cc-by-nc-4.0 license:cc-by-nd-4.0 license:cc-by-sa-4.0 license:cc-by-nc-nd-4.0 license:cc-by-nc-sa-4.0 region:us
{"annotations_creators": ["expert-generated"], "language_creators": ["expert-generated"], "language": ["afr", "af", "aaa", "abc", "ada", "adq", "aeu", "agq", "ags", "ahk", "aia", "ajz", "aka", "ak", "ame", "amh", "am", "amp", "amu", "ann", "aph", "awa", "awb", "azn", "azo", "bag", "bam", "bm", "baw", "bax", "bbk", "bcc", "bce", "bec", "bef", "ben", "bn", "bfd", "bfm", "bfn", "bgf", "bho", "bhs", "bis", "bi", "bjn", "bjr", "bkc", "bkh", "bkm", "bkx", "bob", "bod", "bo", "boz", "bqm", "bra", "brb", "bri", "brv", "bss", "bud", "buo", "bwt", "bwx", "bxa", "bya", "bze", "bzi", "cak", "cbr", "ceb", "cgc", "chd", "chp", "cim", "clo", "cmn", "zh", "cmo", "csw", "cuh", "cuv", "dag", "ddg", "ded", "deu", "de", "dig", "dje", "dmg", "dnw", "dtp", "dtr", "dty", "dug", "eee", "ekm", "enb", "enc", "eng", "en", "ewo", "fas", "fa", "fil", "fli", "fon", "fra", "fr", "fub", "fuh", "gal", "gbj", "gou", "gsw", "guc", "guj", "gu", "guz", "gwc", "hao", "hat", "ht", "hau", "ha", "hbb", "hig", "hil", "hin", "hi", "hla", "hna", "hre", "hro", "idt", "ilo", "ind", "id", "ino", "isu", "ita", "it", "jgo", "jmx", "jpn", "ja", "jra", "kak", "kam", "kan", "kn", "kau", "kr", "kbq", "kbx", "kby", "kek", "ken", "khb", "khm", "km", "kik", "ki", "kin", "rw", "kir", "ky", "kjb", "kmg", "kmr", "ku", "kms", "kmu", "kor", "ko", "kqr", "krr", "ksw", "kur", "ku", "kvt", "kwd", "kwu", "kwx", "kxp", "kyq", "laj", "lan", "lao", "lo", "lbr", "lfa", "lgg", "lgr", "lhm", "lhu", "lkb", "llg", "lmp", "lns", "loh", "lsi", "lts", "lug", "lg", "luy", "lwl", "mai", "mal", "ml", "mam", "mar", "mr", "mdr", "mfh", "mfj", "mgg", "mgm", "mgo", "mgq", "mhx", "miy", "mkz", "mle", "mlk", "mlw", "mmu", "mne", "mnf", "mnw", "mot", "mqj", "mrn", "mry", "msb", "muv", "mve", "mxu", "mya", "my", "myk", "myx", "mzm", "nas", "nco", "nep", "ne", "new", "nge", "ngn", "nhx", "njy", "nla", "nld", "nl", "nlv", "nod", "nsk", "nsn", "nso", "nst", "nuj", "nwe", "nwi", "nxa", "nxl", "nya", "ny", "nyo", "nyu", "nza", "odk", "oji", "oj", "oki", "omw", "ori", "or", "ozm", "pae", "pag", "pan", "pa", "pbt", "pce", "pcg", "pdu", "pea", "pex", "pis", "pkb", "pmf", "pnz", "por", "pt", "psp", "pwg", "qaa", "qub", "quc", "quf", "quz", "qve", "qvh", "qvm", "qvo", "qxh", "rel", "rnl", "ron", "ro", "roo", "rue", "rug", "rus", "ru", "san", "sa", "saq", "sat", "sdk", "sea", "sgd", "shn", "sml", "snk", "snl", "som", "so", "sot", "st", "sox", "spa", "es", "sps", "ssn", "stk", "swa", "sw", "swh", "sxb", "syw", "taj", "tam", "ta", "tbj", "tdb", "tdg", "tdt", "teo", "tet", "tgk", "tg", "tha", "th", "the", "thk", "thl", "thy", "tio", "tkd", "tnl", "tnn", "tnp", "tnt", "tod", "tom", "tpi", "tpl", "tpu", "tsb", "tsn", "tn", "tso", "ts", "tuv", "tuz", "tvs", "udg", "unr", "urd", "ur", "uzb", "uz", "ven", "ve", "vie", "vi", "vif", "war", "wbm", "wbr", "wms", "wni", "wnk", "wtk", "xho", "xh", "xkg", "xmd", "xmg", "xmm", "xog", "xty", "yas", "yav", "ybb", "ybh", "ybi", "ydd", "yea", "yet", "yid", "yi", "yin", "ymp", "zaw", "zho", "zh", "zlm", "zuh", "zul", "zu"], "license": ["cc-by-4.0", "cc-by-nc-4.0", "cc-by-nd-4.0", "cc-by-sa-4.0", "cc-by-nc-nd-4.0", "cc-by-nc-sa-4.0"], "multilinguality": ["multilingual"], "size_categories": ["10K<n<100K"], "source_datasets": ["original"], "task_ids": ["language-modeling"], "pretty_name": "BloomLM", "extra_gated_prompt": "One more step before getting this dataset. This dataset is open access and available only for non-commercial use (except for portions of the dataset labeled with a `cc-by-sa` license). A \"license\" field paired with each of the dataset entries/samples specifies the Creative Commons license for that entry/sample.\n\nThese [Creative Commons licenses](https://creativecommons.org/about/cclicenses/) specify that: \n\n1. You cannot use the dataset for or directed toward commercial advantage or monetary compensation (except for those portions of the dataset labeled specifically with a `cc-by-sa` license. If you would like to ask about commercial uses of this dataset, please [email us](mailto:sj@derivation.co).\n2. Any public, non-commercial use of the data must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. \n3. For those portions of the dataset marked with an ND license, you cannot remix, transform, or build upon the material, and you may not distribute modified material. \n\nIn addition to the above implied by Creative Commons and when clicking \"Access Repository\" below, you agree: \n\n1. Not to use the dataset for any use intended to or which has the effect of harming or enabling discrimination against individuals or groups based on legally protected characteristics or categories, including but not limited to discrimination against Indigenous People as outlined in Articles 2; 13-16; and 31 of the United Nations Declaration on the Rights of Indigenous People, 13 September 2007 and as subsequently amended and revised.\n2. That your *contact information* (email address and username) can be shared with the model authors as well.\n  ", "extra_gated_fields": {"I have read the License and agree with its terms": "checkbox"}}
TAGS
#task_ids-language-modeling #annotations_creators-expert-generated #language_creators-expert-generated #multilinguality-multilingual #size_categories-10K<n<100K #source_datasets-original #language-Afrikaans #language-Afrikaans #language-Ghotuo #language-Ambala Ayta #language-Adangme #language-Adangbe #language-Akeu #language-Aghem #language-Esimbi #language-Akha #language-Arosi #language-Amri Karbi #language-Akan #language-Akan #language-Yanesha' #language-Amharic #language-Amharic #language-Alamblak #language-Guerrero Amuzgo #language-Obolo #language-Athpariya #language-Awadhi #language-Awa (Papua New Guinea) #language-Western Durango Nahuatl #language-Awing #language-Tuki #language-Bambara #language-Bambara #language-Bambili-Bambui #language-Bamun #language-Babanki #language-Southern Balochi #language-Bamenyam #language-Iceve-Maci #language-Benabena #language-Bengali #language-Bengali #language-Bafut #language-Mmen #language-Bunak #language-Bangandu #language-Bhojpuri #language-Buwal #language-Bislama #language-Bislama #language-Banjar #language-Binumarien #language-Baka (Cameroon) #language-Bakoko #language-Kom (Cameroon) #language-Baikeno #language-Aweer #language-Tibetan #language-Tibetan #language-Tiéyaxo Bozo #language-Wumboko #language-Braj #language-Brao #language-Mokpwe #language-Western Bru #language-Akoose #language-Ntcham #language-Terei #language-Bafaw-Balong #language-Bu-Nao Bunu #language-Tairaha #language-Batak #language-Jenaama Bozo #language-Bisu #language-Kaqchikel #language-Cashibo-Cacataibo #language-Cebuano #language-Kagayanen #language-Highland Oaxaca Chontal #language-Chipewyan #language-Cimbrian #language-Lowland Oaxaca Chontal #language-Mandarin Chinese #language-Chinese #language-Central Mnong #language-Swampy Cree #language-Chuka #language-Cuvok #language-Dagbani #language-Fataluku #language-Dedua #language-German #language-German #language-Digo #language-Zarma #language-Upper Kinabatangan #language-Western Dani #language-Kadazan Dusun #language-Lotud #language-Dotyali #language-Duruma #language-E #language-Elip #language-Markweeta #language-En #language-English #language-English #language-Ewondo #language-Persian #language-Persian #language-Filipino #language-Fali #language-Fon #language-French #language-French #language-Adamawa Fulfulde #language-Western Niger Fulfulde #language-Galolen #language-Bodo Gadaba #language-Gavar #language-Swiss German #language-Wayuu #language-Gujarati #language-Gujarati #language-Gusii #language-Gawri #language-Hakö #language-Haitian #language-Haitian #language-Hausa #language-Hausa #language-Huba #language-Kamwe #language-Hiligaynon #language-Hindi #language-Hindi #language-Halia #language-Mina (Cameroon) #language-Hre #language-Haroi #language-Idaté #language-Iloko #language-Indonesian #language-Indonesian #language-Inoke-Yate #language-Isu (Menchum Division) #language-Italian #language-Italian #language-Ngomba #language-Western Juxtlahuaca Mixtec #language-Japanese #language-Japanese #language-Jarai #language-Kalanguya #language-Kamba (Kenya) #language-Kannada #language-Kannada #language-Kanuri #language-Kanuri #language-Kamano #language-Ap Ma #language-Manga Kanuri #language-Kekchí #language-Kenyang #language-Lü #language-Khmer #language-Khmer #language-Kikuyu #language-Kikuyu #language-Kinyarwanda #language-Kinyarwanda #language-Kirghiz #language-Kirghiz #language-Q'anjob'al #language-Kâte #language-Northern Kurdish #language-Kurdish #language-Kamasau #language-Kanite #language-Korean #language-Korean #language-Kimaragang #language-Krung #language-S'gaw Karen #language-Kurdish #language-Lahta Karen #language-Kwaio #language-Kwakum #language-Khirwar #language-Wadiyara Koli #language-Kenga #language-Lango (Uganda) #language-Laru #language-Lao #language-Lao #language-Lohorung #language-Lefa #language-Lugbara #language-Lengo #language-Lhomi #language-Lahu #language-Kabras #language-Lole #language-Limbum #language-Lamnso' #language-Laarim #language-Lashi #language-Tachoni #language-Ganda #language-Ganda #language-Luyia #language-Eastern Lawa #language-Maithili #language-Malayalam #language-Malayalam #language-Mam #language-Marathi #language-Marathi #language-Mandar #language-Matal #language-Mefele #language-Mpumpong #language-Mambae #language-Meta' #language-Malila #language-Maru #language-Ayutla Mixtec #language-Makasae #language-Manambu #language-Ilwana #language-Moloko #language-Mmaala #language-Naba #language-Mundani #language-Mon #language-Barí #language-Mamasa #language-Cheke Holo #language-Mandaya #language-Masbatenyo #language-Muthuvan #language-Marwari (Pakistan) #language-Mada (Cameroon) #language-Burmese #language-Burmese #language-Mamara Senoufo #language-Masaaba #language-Mumuye #language-Naasioi #language-Sibe #language-Nepali (macrolanguage) #language-Nepali (macrolanguage) #language-Newari #language-Ngemba #language-Ngwo #language-Isthmus-Mecayapan Nahuatl #language-Njyem #language-Ngombale #language-Dutch #language-Dutch #language-Orizaba Nahuatl #language-Northern Thai #language-Naskapi #language-Nehan #language-Pedi #language-Tase Naga #language-Nyole #language-Ngwe #language-Southwest Tanna #language-Nauete #language-South Nuaulu #language-Nyanja #language-Nyanja #language-Nyoro #language-Nyungwe #language-Tigon Mbembe #language-Od #language-Ojibwa #language-Ojibwa #language-Okiek #language-South Tairora #language-Oriya (macrolanguage) #language-Oriya (macrolanguage) #language-Koonzime #language-Pagibete #language-Pangasinan #language-Panjabi #language-Panjabi #language-Southern Pashto #language-Ruching Palaung #language-Paniya #language-Kayan #language-Peranakan Indonesian #language-Petats #language-Pijin #language-Pokomo #language-Pamona #language-Pana (Central African Republic) #language-Portuguese #language-Portuguese #language-Philippine Sign Language #language-Gapapaiwa #language-qaa #language-Huallaga Huánuco Quechua #language-K'iche' #language-Lambayeque Quechua #language-Cusco Quechua #language-Eastern Apurímac Quechua #language-Huamalíes-Dos de Mayo Huánuco Quechua #language-Margos-Yarowilca-Lauricocha Quechua #language-Napo Lowland Quechua #language-Panao Huánuco Quechua #language-Rendille #language-Ranglong #language-Romanian #language-Romanian #language-Rotokas #language-Rusyn #language-Roviana #language-Russian #language-Russian #language-Sanskrit #language-Sanskrit #language-Samburu #language-Santali #language-Sos Kundi #language-Semai #language-Surigaonon #language-Shan #language-Central Sama #language-Soninke #language-Sangil #language-Somali #language-Somali #language-Southern Sotho #language-Southern Sotho #language-Swo #language-Spanish #language-Spanish #language-Saposa #language-Waata #language-Arammba #language-Swahili (macrolanguage) #language-Swahili (macrolanguage) #language-Swahili (individual language) #language-Suba #language-Kagate #language-Eastern Tamang #language-Tamil #language-Tamil #language-Tiang #language-Panchpargania #language-Western Tamang #language-Tetun Dili #language-Teso #language-Tetum #language-Tajik #language-Tajik #language-Thai #language-Thai #language-Chitwania Tharu #language-Tharaka #language-Dangaura Tharu #language-Tha #language-Teop #language-Tukudede #language-Lenakel #language-North Tanna #language-Whitesands #language-Tontemboan #language-Toma #language-Tombulu #language-Tok Pisin #language-Tlacoapa Me'phaa #language-Tampuan #language-Tsamai #language-Tswana #language-Tswana #language-Tsonga #language-Tsonga #language-Turkana #language-Turka #language-Taveta #language-Muduga #language-Mundari #language-Urdu #language-Urdu #language-Uzbek #language-Uzbek #language-Venda #language-Venda #language-Vietnamese #language-Vietnamese #language-Vili #language-Waray (Philippines) #language-Wa #language-Wagdi #language-Wambon #language-Ndzwani Comorian #language-Wanukaka #language-Watakataui #language-Xhosa #language-Xhosa #language-Kagoro #language-Mbudum #language-Mengaka #language-Manado Malay #language-Soga #language-Yoloxochitl Mixtec #language-Nugunu (Cameroon) #language-Yangben #language-Yemba #language-Yakha #language-Yamphu #language-Eastern Yiddish #language-Ravula #language-Yetfa #language-Yiddish #language-Yiddish #language-Riang Lai #language-Yamap #language-Mitla Zapotec #language-Chinese #language-Malay (individual language) #language-Tokano #language-Zulu #language-Zulu #license-cc-by-4.0 #license-cc-by-nc-4.0 #license-cc-by-nd-4.0 #license-cc-by-sa-4.0 #license-cc-by-nc-nd-4.0 #license-cc-by-nc-sa-4.0 #region-us 

Table of Contents
-----------------


* Dataset Description


	+ Dataset Summary
	+ Languages
* Dataset Structure


	+ Data Instances
	+ Data Fields
	+ Data Splits


Dataset Description
-------------------


* Homepage: SIL AI
* Point of Contact: SIL AI email
* Source Data: Bloom Library


!logo for Bloom Library !sil-ai logo


Dataset Summary
---------------


Bloom is free, open-source software and an associated website Bloom Library, app, and services developed by SIL International. Bloom’s primary goal is to equip non-dominant language communities and their members to create the literature they want for their community and children. Bloom also serves organizations that help such communities develop literature and education or other aspects of community development.


This version of the Bloom Library data is developed specifically for the language modeling task. It includes data from 364 languages across 31 language families. There is a mean of 32 stories and median of 2 stories per language.


Note: If you speak one of these languages and can help provide feedback or corrections, please let us know!


Note: Although this data was used in the training of the BLOOM model, this dataset only represents a small portion of the data used to train that model. Data from "Bloom Library" was combined with a large number of other datasets to train that model. "Bloom Library" is a project that existed prior to the BLOOM model, and is something separate. All that to say... We were using the "Bloom" name before it was cool.


Languages
---------


Of the 500+ languages listed at URL, there are 363 languages available in this dataset. Here are the corresponding ISO 639-3 codes:


aaa, abc, ada, adq, aeu, afr, agq, ags, ahk, aia, ajz, aka, ame, amh, amp, amu, ann, aph, awa, awb, azn, azo, bag, bam, baw, bax, bbk, bcc, bce, bec, bef, ben, bfd, bfm, bfn, bgf, bho, bhs, bis, bjn, bjr, bkc, bkh, bkm, bkx, bob, bod, boz, bqm, bra, brb, bri, brv, bss, bud, buo, bwt, bwx, bxa, bya, bze, bzi, cak, cbr, ceb, cgc, chd, chp, cim, clo, cmn, cmo, csw, cuh, cuv, dag, ddg, ded, deu, dig, dje, dmg, dnw, dtp, dtr, dty, dug, eee, ekm, enb, enc, eng, ewo, fas, fil, fli, fon, fra, fub, fuh, gal, gbj, gou, gsw, guc, guj, guz, gwc, hao, hat, hau, hbb, hig, hil, hin, hla, hna, hre, hro, idt, ilo, ind, ino, isu, ita, jgo, jmx, jpn, jra, kak, kam, kan, kau, kbq, kbx, kby, kek, ken, khb, khm, kik, kin, kir, kjb, kmg, kmr, kms, kmu, kor, kqr, krr, ksw, kur, kvt, kwd, kwu, kwx, kxp, kyq, laj, lan, lao, lbr, lfa, lgg, lgr, lhm, lhu, lkb, llg, lmp, lns, loh, lsi, lts, lug, luy, lwl, mai, mal, mam, mar, mdr, mfh, mfj, mgg, mgm, mgo, mgq, mhx, miy, mkz, mle, mlk, mlw, mmu, mne, mnf, mnw, mot, mqj, mrn, mry, msb, muv, mve, mxu, mya, myk, myx, mzm, nas, nco, nep, new, nge, ngn, nhx, njy, nla, nld, nlv, nod, nsk, nsn, nso, nst, nuj, nwe, nwi, nxa, nxl, nya, nyo, nyu, nza, odk, oji, oki, omw, ori, ozm, pae, pag, pan, pbt, pce, pcg, pdu, pea, pex, pis, pkb, pmf, pnz, por, psp, pwg, qub, quc, quf, quz, qve, qvh, qvm, qvo, qxh, rel, rnl, ron, roo, rue, rug, rus, san, saq, sat, sdk, sea, sgd, shn, sml, snk, snl, som, sot, sox, spa, sps, ssn, stk, swa, swh, sxb, syw, taj, tam, tbj, tdb, tdg, tdt, teo, tet, tgk, tha, the, thk, thl, thy, tio, tkd, tnl, tnn, tnp, tnt, tod, tom, tpi, tpl, tpu, tsb, tsn, tso, tuv, tuz, tvs, udg, unr, urd, uzb, ven, vie, vif, war, wbm, wbr, wms, wni, wnk, wtk, xho, xkg, xmd, xmg, xmm, xog, xty, yas, yav, ybb, ybh, ybi, ydd, yea, yet, yid, yin, ymp, zaw, zho, zlm, zuh, zul


Dataset Statistics
------------------


Some of the languages included in the dataset just include 1 or a couple of "stories." These are not split between training, validation, and test. For those with higher numbers of available stories we include the following numbers of stories in each split:



Dataset Structure
-----------------


### Data Instances


The examples look like this for Hindi:


This would produce an output:


Whereas if you wish to gather all the text for a language you may use this:


### Data Fields


The metadata fields below are available and the full dataset will be updated with per story metadata soon (in August 2022). As of now a majority of stories have metadata, but some are missing certain fields. In terms of licenses, all stories included in the current release are released under a Creative Commons license (even if the individual story metadata fields are missing).


* text: the text of the story/book, concatenated together from the different pages.
* id: id of the sample
* title: title of the book, e.g. "Going to Buy a Book".
* license: specific license used, e.g. "cc-by-sa" for "Creative Commons, by attribution, share-alike".
* copyright: copyright notice from the original book on URL
* pageCount: page count from the metadata on the original book on URL.
* bookInstanceId: unique ID for each book/translation assigned by Bloom. For example the Hindi version of 'Going to Buy a Book' is 'af86eefd-f69c-4e06-b8eb-e0451853aab9'.
* bookLineage: Unique bookInstanceIDs of *other* Bloom books that this book is in some way based on. For example, the Hindi version in the example above is based on '056B6F11-4A6C-4942-B2BC-8861E62B03B3'. It's quite possible for this to be either empty, or have multiple entries. For example, the book 'Saboo y Jojo' with ID '5b232a5f-561d-4514-afe7-d6ed2f6a940f' is based on two others, ['056B6F11-4A6C-4942-B2BC-8861E62B03B3', '10a6075b-3c4f-40e4-94f3-593497f2793a']
* (coming soon) contentLanguages: Other languages this book may be available in. "Going to Buy a Book" is available in ['eng', 'kan', 'mar', 'pan', 'ben', 'guj', 'hin'] for example.


### Data Splits


All languages include a train, validation, and test split. However, for language having a small number of stories, certain of these splits maybe empty. In such cases, we recommend using any data for testing only or for zero-shot experiments.


Changelog
---------


* 25 August 2022 - add the remaining metadata, change data type of 'pageCount' to int32
* 24 August 2022 - majority of metadata added back in to the filtered/ clean data
* 23 August 2022 - metadata temporarily removed to update to cleaner dataset



### Data Instances


The examples look like this for Hindi:


This would produce an output:


Whereas if you wish to gather all the text for a language you may use this: ### Data Fields


The metadata fields below are available and the full dataset will be updated with per story metadata soon (in August 2022). As of now a majority of stories have metadata, but some are missing certain fields. In terms of licenses, all stories included in the current release are released under a Creative Commons license (even if the individual story metadata fields are missing).


* text: the text of the story/book, concatenated together from the different pages.
* id: id of the sample
* title: title of the book, e.g. "Going to Buy a Book".
* license: specific license used, e.g. "cc-by-sa" for "Creative Commons, by attribution, share-alike".
* copyright: copyright notice from the original book on URL
* pageCount: page count from the metadata on the original book on URL.
* bookInstanceId: unique ID for each book/translation assigned by Bloom. For example the Hindi version of 'Going to Buy a Book' is 'af86eefd-f69c-4e06-b8eb-e0451853aab9'.
* bookLineage: Unique bookInstanceIDs of *other* Bloom books that this book is in some way based on. For example, the Hindi version in the example above is based on '056B6F11-4A6C-4942-B2BC-8861E62B03B3'. It's quite possible for this to be either empty, or have multiple entries. For example, the book 'Saboo y Jojo' with ID '5b232a5f-561d-4514-afe7-d6ed2f6a940f' is based on two others, ['056B6F11-4A6C-4942-B2BC-8861E62B03B3', '10a6075b-3c4f-40e4-94f3-593497f2793a']
* (coming soon) contentLanguages: Other languages this book may be available in. "Going to Buy a Book" is available in ['eng', 'kan', 'mar', 'pan', 'ben', 'guj', 'hin'] for example. ### Data Splits


All languages include a train, validation, and test split. However, for language having a small number of stories, certain of these splits maybe empty. In such cases, we recommend using any data for testing only or for zero-shot experiments.


Changelog
---------


* 25 August 2022 - add the remaining metadata, change data type of 'pageCount' to int32
* 24 August 2022 - majority of metadata added back in to the filtered/ clean data
* 23 August 2022 - metadata temporarily removed to update to cleaner dataset
TAGS
#task_ids-language-modeling #annotations_creators-expert-generated #language_creators-expert-generated #multilinguality-multilingual #size_categories-10K<n<100K #source_datasets-original #language-Afrikaans #language-Afrikaans #language-Ghotuo #language-Ambala Ayta #language-Adangme #language-Adangbe #language-Akeu #language-Aghem #language-Esimbi #language-Akha #language-Arosi #language-Amri Karbi #language-Akan #language-Akan #language-Yanesha' #language-Amharic #language-Amharic #language-Alamblak #language-Guerrero Amuzgo #language-Obolo #language-Athpariya #language-Awadhi #language-Awa (Papua New Guinea) #language-Western Durango Nahuatl #language-Awing #language-Tuki #language-Bambara #language-Bambara #language-Bambili-Bambui #language-Bamun #language-Babanki #language-Southern Balochi #language-Bamenyam #language-Iceve-Maci #language-Benabena #language-Bengali #language-Bengali #language-Bafut #language-Mmen #language-Bunak #language-Bangandu #language-Bhojpuri #language-Buwal #language-Bislama #language-Bislama #language-Banjar #language-Binumarien #language-Baka (Cameroon) #language-Bakoko #language-Kom (Cameroon) #language-Baikeno #language-Aweer #language-Tibetan #language-Tibetan #language-Tiéyaxo Bozo #language-Wumboko #language-Braj #language-Brao #language-Mokpwe #language-Western Bru #language-Akoose #language-Ntcham #language-Terei #language-Bafaw-Balong #language-Bu-Nao Bunu #language-Tairaha #language-Batak #language-Jenaama Bozo #language-Bisu #language-Kaqchikel #language-Cashibo-Cacataibo #language-Cebuano #language-Kagayanen #language-Highland Oaxaca Chontal #language-Chipewyan #language-Cimbrian #language-Lowland Oaxaca Chontal #language-Mandarin Chinese #language-Chinese #language-Central Mnong #language-Swampy Cree #language-Chuka #language-Cuvok #language-Dagbani #language-Fataluku #language-Dedua #language-German #language-German #language-Digo #language-Zarma #language-Upper Kinabatangan #language-Western Dani #language-Kadazan Dusun #language-Lotud #language-Dotyali #language-Duruma #language-E #language-Elip #language-Markweeta #language-En #language-English #language-English #language-Ewondo #language-Persian #language-Persian #language-Filipino #language-Fali #language-Fon #language-French #language-French #language-Adamawa Fulfulde #language-Western Niger Fulfulde #language-Galolen #language-Bodo Gadaba #language-Gavar #language-Swiss German #language-Wayuu #language-Gujarati #language-Gujarati #language-Gusii #language-Gawri #language-Hakö #language-Haitian #language-Haitian #language-Hausa #language-Hausa #language-Huba #language-Kamwe #language-Hiligaynon #language-Hindi #language-Hindi #language-Halia #language-Mina (Cameroon) #language-Hre #language-Haroi #language-Idaté #language-Iloko #language-Indonesian #language-Indonesian #language-Inoke-Yate #language-Isu (Menchum Division) #language-Italian #language-Italian #language-Ngomba #language-Western Juxtlahuaca Mixtec #language-Japanese #language-Japanese #language-Jarai #language-Kalanguya #language-Kamba (Kenya) #language-Kannada #language-Kannada #language-Kanuri #language-Kanuri #language-Kamano #language-Ap Ma #language-Manga Kanuri #language-Kekchí #language-Kenyang #language-Lü #language-Khmer #language-Khmer #language-Kikuyu #language-Kikuyu #language-Kinyarwanda #language-Kinyarwanda #language-Kirghiz #language-Kirghiz #language-Q'anjob'al #language-Kâte #language-Northern Kurdish #language-Kurdish #language-Kamasau #language-Kanite #language-Korean #language-Korean #language-Kimaragang #language-Krung #language-S'gaw Karen #language-Kurdish #language-Lahta Karen #language-Kwaio #language-Kwakum #language-Khirwar #language-Wadiyara Koli #language-Kenga #language-Lango (Uganda) #language-Laru #language-Lao #language-Lao #language-Lohorung #language-Lefa #language-Lugbara #language-Lengo #language-Lhomi #language-Lahu #language-Kabras #language-Lole #language-Limbum #language-Lamnso' #language-Laarim #language-Lashi #language-Tachoni #language-Ganda #language-Ganda #language-Luyia #language-Eastern Lawa #language-Maithili #language-Malayalam #language-Malayalam #language-Mam #language-Marathi #language-Marathi #language-Mandar #language-Matal #language-Mefele #language-Mpumpong #language-Mambae #language-Meta' #language-Malila #language-Maru #language-Ayutla Mixtec #language-Makasae #language-Manambu #language-Ilwana #language-Moloko #language-Mmaala #language-Naba #language-Mundani #language-Mon #language-Barí #language-Mamasa #language-Cheke Holo #language-Mandaya #language-Masbatenyo #language-Muthuvan #language-Marwari (Pakistan) #language-Mada (Cameroon) #language-Burmese #language-Burmese #language-Mamara Senoufo #language-Masaaba #language-Mumuye #language-Naasioi #language-Sibe #language-Nepali (macrolanguage) #language-Nepali (macrolanguage) #language-Newari #language-Ngemba #language-Ngwo #language-Isthmus-Mecayapan Nahuatl #language-Njyem #language-Ngombale #language-Dutch #language-Dutch #language-Orizaba Nahuatl #language-Northern Thai #language-Naskapi #language-Nehan #language-Pedi #language-Tase Naga #language-Nyole #language-Ngwe #language-Southwest Tanna #language-Nauete #language-South Nuaulu #language-Nyanja #language-Nyanja #language-Nyoro #language-Nyungwe #language-Tigon Mbembe #language-Od #language-Ojibwa #language-Ojibwa #language-Okiek #language-South Tairora #language-Oriya (macrolanguage) #language-Oriya (macrolanguage) #language-Koonzime #language-Pagibete #language-Pangasinan #language-Panjabi #language-Panjabi #language-Southern Pashto #language-Ruching Palaung #language-Paniya #language-Kayan #language-Peranakan Indonesian #language-Petats #language-Pijin #language-Pokomo #language-Pamona #language-Pana (Central African Republic) #language-Portuguese #language-Portuguese #language-Philippine Sign Language #language-Gapapaiwa #language-qaa #language-Huallaga Huánuco Quechua #language-K'iche' #language-Lambayeque Quechua #language-Cusco Quechua #language-Eastern Apurímac Quechua #language-Huamalíes-Dos de Mayo Huánuco Quechua #language-Margos-Yarowilca-Lauricocha Quechua #language-Napo Lowland Quechua #language-Panao Huánuco Quechua #language-Rendille #language-Ranglong #language-Romanian #language-Romanian #language-Rotokas #language-Rusyn #language-Roviana #language-Russian #language-Russian #language-Sanskrit #language-Sanskrit #language-Samburu #language-Santali #language-Sos Kundi #language-Semai #language-Surigaonon #language-Shan #language-Central Sama #language-Soninke #language-Sangil #language-Somali #language-Somali #language-Southern Sotho #language-Southern Sotho #language-Swo #language-Spanish #language-Spanish #language-Saposa #language-Waata #language-Arammba #language-Swahili (macrolanguage) #language-Swahili (macrolanguage) #language-Swahili (individual language) #language-Suba #language-Kagate #language-Eastern Tamang #language-Tamil #language-Tamil #language-Tiang #language-Panchpargania #language-Western Tamang #language-Tetun Dili #language-Teso #language-Tetum #language-Tajik #language-Tajik #language-Thai #language-Thai #language-Chitwania Tharu #language-Tharaka #language-Dangaura Tharu #language-Tha #language-Teop #language-Tukudede #language-Lenakel #language-North Tanna #language-Whitesands #language-Tontemboan #language-Toma #language-Tombulu #language-Tok Pisin #language-Tlacoapa Me'phaa #language-Tampuan #language-Tsamai #language-Tswana #language-Tswana #language-Tsonga #language-Tsonga #language-Turkana #language-Turka #language-Taveta #language-Muduga #language-Mundari #language-Urdu #language-Urdu #language-Uzbek #language-Uzbek #language-Venda #language-Venda #language-Vietnamese #language-Vietnamese #language-Vili #language-Waray (Philippines) #language-Wa #language-Wagdi #language-Wambon #language-Ndzwani Comorian #language-Wanukaka #language-Watakataui #language-Xhosa #language-Xhosa #language-Kagoro #language-Mbudum #language-Mengaka #language-Manado Malay #language-Soga #language-Yoloxochitl Mixtec #language-Nugunu (Cameroon) #language-Yangben #language-Yemba #language-Yakha #language-Yamphu #language-Eastern Yiddish #language-Ravula #language-Yetfa #language-Yiddish #language-Yiddish #language-Riang Lai #language-Yamap #language-Mitla Zapotec #language-Chinese #language-Malay (individual language) #language-Tokano #language-Zulu #language-Zulu #license-cc-by-4.0 #license-cc-by-nc-4.0 #license-cc-by-nd-4.0 #license-cc-by-sa-4.0 #license-cc-by-nc-nd-4.0 #license-cc-by-nc-sa-4.0 #region-us 
 ### Data Instances


The examples look like this for Hindi:


This would produce an output:


Whereas if you wish to gather all the text for a language you may use this: ### Data Fields


The metadata fields below are available and the full dataset will be updated with per story metadata soon (in August 2022). As of now a majority of stories have metadata, but some are missing certain fields. In terms of licenses, all stories included in the current release are released under a Creative Commons license (even if the individual story metadata fields are missing).


* text: the text of the story/book, concatenated together from the different pages.
* id: id of the sample
* title: title of the book, e.g. "Going to Buy a Book".
* license: specific license used, e.g. "cc-by-sa" for "Creative Commons, by attribution, share-alike".
* copyright: copyright notice from the original book on URL
* pageCount: page count from the metadata on the original book on URL.
* bookInstanceId: unique ID for each book/translation assigned by Bloom. For example the Hindi version of 'Going to Buy a Book' is 'af86eefd-f69c-4e06-b8eb-e0451853aab9'.
* bookLineage: Unique bookInstanceIDs of *other* Bloom books that this book is in some way based on. For example, the Hindi version in the example above is based on '056B6F11-4A6C-4942-B2BC-8861E62B03B3'. It's quite possible for this to be either empty, or have multiple entries. For example, the book 'Saboo y Jojo' with ID '5b232a5f-561d-4514-afe7-d6ed2f6a940f' is based on two others, ['056B6F11-4A6C-4942-B2BC-8861E62B03B3', '10a6075b-3c4f-40e4-94f3-593497f2793a']
* (coming soon) contentLanguages: Other languages this book may be available in. "Going to Buy a Book" is available in ['eng', 'kan', 'mar', 'pan', 'ben', 'guj', 'hin'] for example. ### Data Splits


All languages include a train, validation, and test split. However, for language having a small number of stories, certain of these splits maybe empty. In such cases, we recommend using any data for testing only or for zero-shot experiments.


Changelog
---------


* 25 August 2022 - add the remaining metadata, change data type of 'pageCount' to int32
* 24 August 2022 - majority of metadata added back in to the filtered/ clean data
* 23 August 2022 - metadata temporarily removed to update to cleaner dataset
passage:  passage: TAGS
#task_ids-language-modeling #annotations_creators-expert-generated #language_creators-expert-generated #multilinguality-multilingual #size_categories-10K<n<100K #source_datasets-original #language-Afrikaans #language-Afrikaans #language-Ghotuo #language-Ambala Ayta #language-Adangme #language-Adangbe #language-Akeu #language-Aghem #language-Esimbi #language-Akha #language-Arosi #language-Amri Karbi #language-Akan #language-Akan #language-Yanesha' #language-Amharic #language-Amharic #language-Alamblak #language-Guerrero Amuzgo #language-Obolo #language-Athpariya #language-Awadhi #language-Awa (Papua New Guinea) #language-Western Durango Nahuatl #language-Awing #language-Tuki #language-Bambara #language-Bambara #language-Bambili-Bambui #language-Bamun #language-Babanki #language-Southern Balochi #language-Bamenyam #language-Iceve-Maci #language-Benabena #language-Bengali #language-Bengali #language-Bafut #language-Mmen #language-Bunak #language-Bangandu #language-Bhojpuri #language-Buwal #language-Bislama #language-Bislama #language-Banjar #language-Binumarien #language-Baka (Cameroon) #language-Bakoko #language-Kom (Cameroon) #language-Baikeno #language-Aweer #language-Tibetan #language-Tibetan #language-Tiéyaxo Bozo #language-Wumboko #language-Braj #language-Brao #language-Mokpwe #language-Western Bru #language-Akoose #language-Ntcham #language-Terei #language-Bafaw-Balong #language-Bu-Nao Bunu #language-Tairaha #language-Batak #language-Jenaama Bozo #language-Bisu #language-Kaqchikel #language-Cashibo-Cacataibo #language-Cebuano #language-Kagayanen #language-Highland Oaxaca Chontal #language-Chipewyan #language-Cimbrian #language-Lowland Oaxaca Chontal #language-Mandarin Chinese #language-Chinese #language-Central Mnong #language-Swampy Cree #language-Chuka #language-Cuvok #language-Dagbani #language-Fataluku #language-Dedua #language-German #language-German #language-Digo #language-Zarma #language-Upper Kinabatangan #language-Western Dani #language-Kadazan Dusun #language-Lotud #language-Dotyali #language-Duruma #language-E #language-Elip #language-Markweeta #language-En #language-English #language-English #language-Ewondo #language-Persian #language-Persian #language-Filipino #language-Fali #language-Fon #language-French #language-French #language-Adamawa Fulfulde #language-Western Niger Fulfulde #language-Galolen #language-Bodo Gadaba #language-Gavar #language-Swiss German #language-Wayuu #language-Gujarati #language-Gujarati #language-Gusii #language-Gawri #language-Hakö #language-Haitian #language-Haitian #language-Hausa #language-Hausa #language-Huba #language-Kamwe #language-Hiligaynon #language-Hindi #language-Hindi #language-Halia #language-Mina (Cameroon) #language-Hre #language-Haroi #language-Idaté #language-Iloko #language-Indonesian #language-Indonesian #language-Inoke-Yate #language-Isu (Menchum Division) #language-Italian #language-Italian #language-Ngomba #language-Western Juxtlahuaca Mixtec #language-Japanese #language-Japanese #language-Jarai #language-Kalanguya #language-Kamba (Kenya) #language-Kannada #language-Kannada #language-Kanuri #language-Kanuri #language-Kamano #language-Ap Ma #language-Manga Kanuri #language-Kekchí #language-Kenyang #language-Lü #language-Khmer #language-Khmer #language-Kikuyu #language-Kikuyu #language-Kinyarwanda #language-Kinyarwanda #language-Kirghiz #language-Kirghiz #language-Q'anjob'al #language-Kâte #language-Northern Kurdish #language-Kurdish #language-Kamasau #language-Kanite #language-Korean #language-Korean #language-Kimaragang #language-Krung #language-S'gaw Karen #language-Kurdish #language-Lahta Karen #language-Kwaio #language-Kwakum #language-Khirwar #language-Wadiyara Koli #language-Kenga #language-Lango (Uganda) #language-Laru #language-Lao #language-Lao #language-Lohorung #language-Lefa #language-Lugbara #language-Lengo #language-Lhomi #language-Lahu #language-Kabras #language-Lole #language-Limbum #language-Lamnso' #language-Laarim #language-Lashi #language-Tachoni #language-Ganda #language-Ganda #language-Luyia #language-Eastern Lawa #language-Maithili #language-Malayalam #language-Malayalam #language-Mam #language-Marathi #language-Marathi #language-Mandar #language-Matal #language-Mefele #language-Mpumpong #language-Mambae #language-Meta' #language-Malila #language-Maru #language-Ayutla Mixtec #language-Makasae #language-Manambu #language-Ilwana #language-Moloko #language-Mmaala #language-Naba #language-Mundani #language-Mon #language-Barí #language-Mamasa #language-Cheke Holo #language-Mandaya #language-Masbatenyo #language-Muthuvan #language-Marwari (Pakistan) #language-Mada (Cameroon) #language-Burmese #language-Burmese #language-Mamara Senoufo #language-Masaaba #language-Mumuye #language-Naasioi #language-Sibe #language-Nepali (macrolanguage) #language-Nepali (macrolanguage) #language-Newari #language-Ngemba #language-Ngwo #language-Isthmus-Mecayapan Nahuatl #language-Njyem #language-Ngombale #language-Dutch #language-Dutch #language-Orizaba Nahuatl #language-Northern Thai #language-Naskapi #language-Nehan #language-Pedi #language-Tase Naga #language-Nyole #language-Ngwe #language-Southwest Tanna #language-Nauete #language-South Nuaulu #language-Nyanja #language-Nyanja #language-Nyoro #language-Nyungwe #language-Tigon Mbembe #language-Od #language-Ojibwa #language-Ojibwa #language-Okiek #language-South Tairora #language-Oriya (macrolanguage) #language-Oriya (macrolanguage) #language-Koonzime #language-Pagibete #language-Pangasinan #language-Panjabi #language-Panjabi #language-Southern Pashto #language-Ruching Palaung #language-Paniya #language-Kayan #language-Peranakan Indonesian #language-Petats #language-Pijin #language-Pokomo #language-Pamona #language-Pana (Central African Republic) #language-Portuguese #language-Portuguese #language-Philippine Sign Language #language-Gapapaiwa #language-qaa #language-Huallaga Huánuco Quechua #language-K'iche' #language-Lambayeque Quechua #language-Cusco Quechua #language-Eastern Apurímac Quechua #language-Huamalíes-Dos de Mayo Huánuco Quechua #language-Margos-Yarowilca-Lauricocha Quechua #language-Napo Lowland Quechua #language-Panao Huánuco Quechua #language-Rendille #language-Ranglong #language-Romanian #language-Romanian #language-Rotokas #language-Rusyn #language-Roviana #language-Russian #language-Russian #language-Sanskrit #language-Sanskrit #language-Samburu #language-Santali #language-Sos Kundi #language-Semai #language-Surigaonon #language-Shan #language-Central Sama #language-Soninke #language-Sangil #language-Somali #language-Somali #language-Southern Sotho #language-Southern Sotho #language-Swo #language-Spanish #language-Spanish #language-Saposa #language-Waata #language-Arammba #language-Swahili (macrolanguage) #language-Swahili (macrolanguage) #language-Swahili (individual language) #language-Suba #language-Kagate #language-Eastern Tamang #language-Tamil #language-Tamil #language-Tiang #language-Panchpargania #language-Western Tamang #language-Tetun Dili #language-Teso #language-Tetum #language-Tajik #language-Tajik #language-Thai #language-Thai #language-Chitwania Tharu #language-Tharaka #language-Dangaura Tharu #language-Tha #language-Teop #language-Tukudede #language-Lenakel #language-North Tanna #language-Whitesands #language-Tontemboan #language-Toma #language-Tombulu #language-Tok Pisin #language-Tlacoapa Me'phaa #language-Tampuan #language-Tsamai #language-Tswana #language-Tswana #language-Tsonga #language-Tsonga #language-Turkana #language-Turka #language-Taveta #language-Muduga #language-Mundari #language-Urdu #language-Urdu #language-Uzbek #language-Uzbek #language-Venda #language-Venda #language-Vietnamese #language-Vietnamese #language-Vili #language-Waray (Philippines) #language-Wa #language-Wagdi #language-Wambon #language-Ndzwani Comorian #language-Wanukaka #language-Watakataui #language-Xhosa #language-Xhosa #language-Kagoro #language-Mbudum #language-Mengaka #language-Manado Malay #language-Soga #language-Yoloxochitl Mixtec #language-Nugunu (Cameroon) #language-Yangben #language-Yemba #language-Yakha #language-Yamphu #language-Eastern Yiddish #language-Ravula #language-Yetfa #language-Yiddish #language-Yiddish #language-Riang Lai #language-Yamap #language-Mitla Zapotec #language-Chinese #language-Malay (individual language) #language-Tokano #language-Zulu #language-Zulu #license-cc-by-4.0 #license-cc-by-nc-4.0 #license-cc-by-nd-4.0 #license-cc-by-sa-4.0 #license-cc-by-nc-nd-4.0 #license-cc-by-nc-sa-4.0 #region-us 
### Data Instances


The examples look like this for Hindi:


This would produce an output:


Whereas if you wish to gather all the text for a language you may use this: