
# Dataset Card for wili_2018

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

## Dataset Description

- **Homepage:** https://zenodo.org/record/841984
- **Repository:** [Needs More Information]
- **Paper:** https://arxiv.org/pdf/1801.07779
- **Leaderboard:** [Needs More Information]
- **Point of Contact:** Thoma, Martin (Email: info@martin-thoma.de)

### Dataset Summary

WiLI-2018, the Wikipedia language identification benchmark dataset, contains 235000 paragraphs of 235 languages. The dataset is balanced and a train-test split is provided.

### Supported Tasks and Leaderboards

[Needs More Information]

### Languages

235 Different Languages

## Dataset Structure

### Data Instances

```
{
    'label': 207,
    'sentence': 'Ti Turkia ket maysa a demokrata, sekular, unitario, batay-linteg a republika nga addaan ti taga-ugma a tinawtawid a kultura. Ti Turkia ket umadadu a naipatipon iti Laud babaen ti panagkameng kadagiti organisasion a kas ti Konsilo iti Europa, NATO, OECD, OSCE ken ti G-20 a dagiti kangrunaan nga ekonomia. Ti Turkia ket nangrugi a nakitulag ti napno a panagkameng iti Kappon ti Europa idi 2005, nga isu ket maysa idin a kumaduaan a kameng iti Europeano a Komunidad ti Ekonomia manipud idi 1963 ken nakadanon ti maysa a tulagan ti kappon ti aduana idi 1995. Ti Turkia ket nagtaraken iti asideg a kultural, politikal, ekonomiko ken industria a panakibiang iti Tengnga a Daya, dagiti Turko nga estado iti Tengnga nga Asia ken dagiti pagilian ti Aprika babaen ti panagkameng kadagiti organisasion a kas ti Turko a Konsilo, Nagsaupan nga Administrasion iti Turko nga Arte ken Kultura, Organisasion iti Islamiko a Panagtitinnulong ken ti Organisasion ti Ekonomiko a Panagtitinnulong.'
}
```

### Data Fields

[Needs More Information]

### Data Splits

175000 lines of text each for train and test data.

## Dataset Creation

### Curation Rationale

[Needs More Information]

### Source Data

#### Initial Data Collection and Normalization

[Needs More Information]

#### Who are the source language producers?

[Needs More Information]

### Annotations

#### Annotation process

[Needs More Information]

#### Who are the annotators?

[Needs More Information]

### Personal and Sensitive Information

[Needs More Information]

## Considerations for Using the Data

### Social Impact of Dataset

[Needs More Information]

### Discussion of Biases

[Needs More Information]

### Other Known Limitations

[Needs More Information]

## Additional Information

### Dataset Curators

The dataset was initially created by Thomas Martin

### Licensing Information

ODC Open Database License v1.0

### Citation Information

```
@dataset{thoma_martin_2018_841984,
  author       = {Thoma, Martin},
  title        = {{WiLI-2018 - Wikipedia Language Identification database}},
  month        = jan,
  year         = 2018,
  publisher    = {Zenodo},
  version      = {1.0.0},
  doi          = {10.5281/zenodo.841984},
  url          = {https://doi.org/10.5281/zenodo.841984}
}
```

### Contributions

Thanks to [@Shubhambindal2017](https://github.com/Shubhambindal2017) for adding this dataset.
task_categories:text-classification annotations_creators:no-annotation language_creators:found multilinguality:multilingual size_categories:100K<n<1M source_datasets:original language:ace language:af language:als language:am language:an language:ang language:ar language:arz language:as language:ast language:av language:ay language:az language:azb language:ba language:bar language:bcl language:be language:bg language:bho language:bjn language:bn language:bo language:bpy language:br language:bs language:bxr language:ca language:cbk language:cdo language:ce language:ceb language:chr language:ckb language:co language:crh language:cs language:csb language:cv language:cy language:da language:de language:diq language:dsb language:dty language:dv language:egl language:el language:en language:eo language:es language:et language:eu language:ext language:fa language:fi language:fo language:fr language:frp language:fur language:fy language:ga language:gag language:gd language:gl language:glk language:gn language:gu language:gv language:ha language:hak language:he language:hi language:hif language:hr language:hsb language:ht language:hu language:hy language:ia language:id language:ie language:ig language:ilo language:io language:is language:it language:ja language:jam language:jbo language:jv language:ka language:kaa language:kab language:kbd language:kk language:km language:kn language:ko language:koi language:kok language:krc language:ksh language:ku language:kv language:kw language:ky language:la language:lad language:lb language:lez language:lg language:li language:lij language:lmo language:ln language:lo language:lrc language:lt language:ltg language:lv language:lzh language:mai language:map language:mdf language:mg language:mhr language:mi language:min language:mk language:ml language:mn language:mr language:mrj language:ms language:mt language:mwl language:my language:myv language:mzn language:nan language:nap language:nb language:nci language:nds language:ne language:new language:nl language:nn language:nrm language:nso language:nv language:oc language:olo language:om language:or language:os language:pa language:pag language:pam language:pap language:pcd language:pdc language:pfl language:pl language:pnb language:ps language:pt language:qu language:rm language:ro language:roa language:ru language:rue language:rup language:rw language:sa language:sah language:sc language:scn language:sco language:sd language:sgs language:sh language:si language:sk language:sl language:sme language:sn language:so language:sq language:sr language:srn language:stq language:su language:sv language:sw language:szl language:ta language:tcy language:te language:tet language:tg language:th language:tk language:tl language:tn language:to language:tr language:tt language:tyv language:udm language:ug language:uk language:ur language:uz language:vec language:vep language:vi language:vls language:vo language:vro language:wa language:war language:wo language:wuu language:xh language:xmf language:yi language:yo language:zea language:zh license:odbl language-identification arxiv:1801.07779 region:us
{"annotations_creators": ["no-annotation"], "language_creators": ["found"], "language": ["ace", "af", "als", "am", "an", "ang", "ar", "arz", "as", "ast", "av", "ay", "az", "azb", "ba", "bar", "bcl", "be", "bg", "bho", "bjn", "bn", "bo", "bpy", "br", "bs", "bxr", "ca", "cbk", "cdo", "ce", "ceb", "chr", "ckb", "co", "crh", "cs", "csb", "cv", "cy", "da", "de", "diq", "dsb", "dty", "dv", "egl", "el", "en", "eo", "es", "et", "eu", "ext", "fa", "fi", "fo", "fr", "frp", "fur", "fy", "ga", "gag", "gd", "gl", "glk", "gn", "gu", "gv", "ha", "hak", "he", "hi", "hif", "hr", "hsb", "ht", "hu", "hy", "ia", "id", "ie", "ig", "ilo", "io", "is", "it", "ja", "jam", "jbo", "jv", "ka", "kaa", "kab", "kbd", "kk", "km", "kn", "ko", "koi", "kok", "krc", "ksh", "ku", "kv", "kw", "ky", "la", "lad", "lb", "lez", "lg", "li", "lij", "lmo", "ln", "lo", "lrc", "lt", "ltg", "lv", "lzh", "mai", "map", "mdf", "mg", "mhr", "mi", "min", "mk", "ml", "mn", "mr", "mrj", "ms", "mt", "mwl", "my", "myv", "mzn", "nan", "nap", "nb", "nci", "nds", "ne", "new", "nl", "nn", "nrm", "nso", "nv", "oc", "olo", "om", "or", "os", "pa", "pag", "pam", "pap", "pcd", "pdc", "pfl", "pl", "pnb", "ps", "pt", "qu", "rm", "ro", "roa", "ru", "rue", "rup", "rw", "sa", "sah", "sc", "scn", "sco", "sd", "sgs", "sh", "si", "sk", "sl", "sme", "sn", "so", "sq", "sr", "srn", "stq", "su", "sv", "sw", "szl", "ta", "tcy", "te", "tet", "tg", "th", "tk", "tl", "tn", "to", "tr", "tt", "tyv", "udm", "ug", "uk", "ur", "uz", "vec", "vep", "vi", "vls", "vo", "vro", "wa", "war", "wo", "wuu", "xh", "xmf", "yi", "yo", "zea", "zh"], "license": ["odbl"], "multilinguality": ["multilingual"], "size_categories": ["100K<n<1M"], "source_datasets": ["original"], "task_categories": ["text-classification"], "task_ids": [], "paperswithcode_id": "wili-2018", "pretty_name": "Wili2018", "language_bcp47": ["be-tarask", "map-bms", "nds-nl", "roa-tara", "zh-yue"], "tags": ["language-identification"], "dataset_info": {"features": [{"name": "sentence", "dtype": "string"}, {"name": "label", "dtype": {"class_label": {"names": {"0": "cdo", "1": "glk", "2": "jam", "3": "lug", "4": "san", "5": "rue", "6": "wol", "7": "new", "8": "mwl", "9": "bre", "10": "ara", "11": "hye", "12": "xmf", "13": "ext", "14": "cor", "15": "yor", "16": "div", "17": "asm", "18": "lat", "19": "cym", "20": "hif", "21": "ace", "22": "kbd", "23": "tgk", "24": "rus", "25": "nso", "26": "mya", "27": "msa", "28": "ava", "29": "cbk", "30": "urd", "31": "deu", "32": "swa", "33": "pus", "34": "bxr", "35": "udm", "36": "csb", "37": "yid", "38": "vro", "39": "por", "40": "pdc", "41": "eng", "42": "tha", "43": "hat", "44": "lmo", "45": "pag", "46": "jav", "47": "chv", "48": "nan", "49": "sco", "50": "kat", "51": "bho", "52": "bos", "53": "kok", "54": "oss", "55": "mri", "56": "fry", "57": "cat", "58": "azb", "59": "kin", "60": "hin", "61": "sna", "62": "dan", "63": "egl", "64": "mkd", "65": "ron", "66": "bul", "67": "hrv", "68": "som", "69": "pam", "70": "nav", "71": "ksh", "72": "nci", "73": "khm", "74": "sgs", "75": "srn", "76": "bar", "77": "cos", "78": "ckb", "79": "pfl", "80": "arz", "81": "roa-tara", "82": "fra", "83": "mai", "84": "zh-yue", "85": "guj", "86": "fin", "87": "kir", "88": "vol", "89": "hau", "90": "afr", "91": "uig", "92": "lao", "93": "swe", "94": "slv", "95": "kor", "96": "szl", "97": "srp", "98": "dty", "99": "nrm", "100": "dsb", "101": "ind", "102": "wln", "103": "pnb", "104": "ukr", "105": "bpy", "106": "vie", "107": "tur", "108": "aym", "109": "lit", "110": "zea", "111": "pol", "112": "est", "113": "scn", "114": "vls", "115": "stq", "116": "gag", "117": "grn", "118": "kaz", "119": "ben", "120": "pcd", "121": "bjn", "122": "krc", "123": "amh", "124": "diq", "125": "ltz", "126": "ita", "127": "kab", "128": "bel", "129": "ang", "130": "mhr", "131": "che", "132": "koi", "133": "glv", "134": "ido", "135": "fao", "136": "bak", "137": "isl", "138": "bcl", "139": "tet", "140": "jpn", "141": "kur", "142": "map-bms", "143": "tyv", "144": "olo", "145": "arg", "146": "ori", "147": "lim", "148": "tel", "149": "lin", "150": "roh", "151": "sqi", "152": "xho", "153": "mlg", "154": "fas", "155": "hbs", "156": "tam", "157": "aze", "158": "lad", "159": "nob", "160": "sin", "161": "gla", "162": "nap", "163": "snd", "164": "ast", "165": "mal", "166": "mdf", "167": "tsn", "168": "nds", "169": "tgl", "170": "nno", "171": "sun", "172": "lzh", "173": "jbo", "174": "crh", "175": "pap", "176": "oci", "177": "hak", "178": "uzb", "179": "zho", "180": "hsb", "181": "sme", "182": "mlt", "183": "vep", "184": "lez", "185": "nld", "186": "nds-nl", "187": "mrj", "188": "spa", "189": "ceb", "190": "ina", "191": "heb", "192": "hun", "193": "que", "194": "kaa", "195": "mar", "196": "vec", "197": "frp", "198": "ell", "199": "sah", "200": "eus", "201": "ces", "202": "slk", "203": "chr", "204": "lij", "205": "nep", "206": "srd", "207": "ilo", "208": "be-tarask", "209": "bod", "210": "orm", "211": "war", "212": "glg", "213": "mon", "214": "gle", "215": "min", "216": "ibo", "217": "ile", "218": "epo", "219": "lav", "220": "lrc", "221": "als", "222": "mzn", "223": "rup", "224": "fur", "225": "tat", "226": "myv", "227": "pan", "228": "ton", "229": "kom", "230": "wuu", "231": "tcy", "232": "tuk", "233": "kan", "234": "ltg"}}}}], "config_name": "WiLI-2018 dataset", "splits": [{"name": "train", "num_bytes": 65408201, "num_examples": 117500}, {"name": "test", "num_bytes": 66491260, "num_examples": 117500}], "download_size": 130516351, "dataset_size": 131899461}}
TAGS
#task_categories-text-classification #annotations_creators-no-annotation #language_creators-found #multilinguality-multilingual #size_categories-100K<n<1M #source_datasets-original #language-Achinese #language-Afrikaans #language-Tosk Albanian #language-Amharic #language-Aragonese #language-Old English (ca. 450-1100) #language-Arabic #language-Egyptian Arabic #language-Assamese #language-Asturian #language-Avaric #language-Aymara #language-Azerbaijani #language-South Azerbaijani #language-Bashkir #language-Bavarian #language-Central Bikol #language-Belarusian #language-Bulgarian #language-Bhojpuri #language-Banjar #language-Bengali #language-Tibetan #language-Bishnupriya #language-Breton #language-Bosnian #language-Russia Buriat #language-Catalan #language-Chavacano #language-Min Dong Chinese #language-Chechen #language-Cebuano #language-Cherokee #language-Central Kurdish #language-Corsican #language-Crimean Tatar #language-Czech #language-Kashubian #language-Chuvash #language-Welsh #language-Danish #language-German #language-Dimli (individual language) #language-Lower Sorbian #language-Dotyali #language-Dhivehi #language-Emilian #language-Modern Greek (1453-) #language-English #language-Esperanto #language-Spanish #language-Estonian #language-Basque #language-Extremaduran #language-Persian #language-Finnish #language-Faroese #language-French #language-Arpitan #language-Friulian #language-Western Frisian #language-Irish #language-Gagauz #language-Scottish Gaelic #language-Galician #language-Gilaki #language-Guarani #language-Gujarati #language-Manx #language-Hausa #language-Hakka Chinese #language-Hebrew #language-Hindi #language-Fiji Hindi #language-Croatian #language-Upper Sorbian #language-Haitian #language-Hungarian #language-Armenian #language-Interlingua (International Auxiliary Language Association) #language-Indonesian #language-Interlingue #language-Igbo #language-Iloko #language-Ido #language-Icelandic #language-Italian #language-Japanese #language-Jamaican Creole English #language-Lojban #language-Javanese #language-Georgian #language-Kara-Kalpak #language-Kabyle #language-Kabardian #language-Kazakh #language-Khmer #language-Kannada #language-Korean #language-Komi-Permyak #language-Konkani (macrolanguage) #language-Karachay-Balkar #language-Kölsch #language-Kurdish #language-Komi #language-Cornish #language-Kirghiz #language-Latin #language-Ladino #language-Luxembourgish #language-Lezghian #language-Ganda #language-Limburgan #language-Ligurian #language-Lombard #language-Lingala #language-Lao #language-Northern Luri #language-Lithuanian #language-Latgalian #language-Latvian #language-Literary Chinese #language-Maithili #language-map #language-Moksha #language-Malagasy #language-Eastern Mari #language-Maori #language-Minangkabau #language-Macedonian #language-Malayalam #language-Mongolian #language-Marathi #language-Western Mari #language-Malay (macrolanguage) #language-Maltese #language-Mirandese #language-Burmese #language-Erzya #language-Mazanderani #language-Min Nan Chinese #language-Neapolitan #language-Norwegian Bokmål #language-Classical Nahuatl #language-Low German #language-Nepali (macrolanguage) #language-Newari #language-Dutch #language-Norwegian Nynorsk #language-Narom #language-Pedi #language-Navajo #language-Occitan (post 1500) #language-Livvi #language-Oromo #language-Oriya (macrolanguage) #language-Ossetian #language-Panjabi #language-Pangasinan #language-Pampanga #language-Papiamento #language-Picard #language-Pennsylvania German #language-Pfaelzisch #language-Polish #language-Western Panjabi #language-Pushto #language-Portuguese #language-Quechua #language-Romansh #language-Romanian #language-roa #language-Russian #language-Rusyn #language-Macedo-Romanian #language-Kinyarwanda #language-Sanskrit #language-Yakut #language-Sardinian #language-Sicilian #language-Scots #language-Sindhi #language-Samogitian #language-Serbo-Croatian #language-Sinhala #language-Slovak #language-Slovenian #language-Northern Sami #language-Shona #language-Somali #language-Albanian #language-Serbian #language-Sranan Tongo #language-Saterfriesisch #language-Sundanese #language-Swedish #language-Swahili (macrolanguage) #language-Silesian #language-Tamil #language-Tulu #language-Telugu #language-Tetum #language-Tajik #language-Thai #language-Turkmen #language-Tagalog #language-Tswana #language-Tonga (Tonga Islands) #language-Turkish #language-Tatar #language-Tuvinian #language-Udmurt #language-Uighur #language-Ukrainian #language-Urdu #language-Uzbek #language-Venetian #language-Veps #language-Vietnamese #language-Vlaams #language-Volapük #language-Võro #language-Walloon #language-Waray (Philippines) #language-Wolof #language-Wu Chinese #language-Xhosa #language-Mingrelian #language-Yiddish #language-Yoruba #language-Zeeuws #language-Chinese #license-odbl #language-identification #arxiv-1801.07779 #region-us 


# Dataset Card for wili_2018

## Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information
  - Contributions

## Dataset Description

- Homepage: URL
- Repository: 
- Paper: URL
- Leaderboard: 
- Point of Contact: Thoma, Martin (Email: info@URL)

### Dataset Summary

WiLI-2018, the Wikipedia language identification benchmark dataset, contains 235000 paragraphs of 235 languages. The dataset is balanced and a train-test split is provided.

### Supported Tasks and Leaderboards



### Languages

235 Different Languages

## Dataset Structure

### Data Instances



### Data Fields



### Data Splits

175000 lines of text each for train and test data.

## Dataset Creation

### Curation Rationale



### Source Data

#### Initial Data Collection and Normalization



#### Who are the source language producers?



### Annotations

#### Annotation process



#### Who are the annotators?



### Personal and Sensitive Information



## Considerations for Using the Data

### Social Impact of Dataset



### Discussion of Biases



### Other Known Limitations



## Additional Information

### Dataset Curators

The dataset was initially created by Thomas Martin

### Licensing Information

ODC Open Database License v1.0





### Contributions

Thanks to @Shubhambindal2017 for adding this dataset.
# Dataset Card for wili_2018 ## Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information
  - Contributions ## Dataset Description

- Homepage: URL
- Repository: 
- Paper: URL
- Leaderboard: 
- Point of Contact: Thoma, Martin (Email: info@URL) ### Dataset Summary

WiLI-2018, the Wikipedia language identification benchmark dataset, contains 235000 paragraphs of 235 languages. The dataset is balanced and a train-test split is provided. ### Supported Tasks and Leaderboards ### Languages

235 Different Languages ## Dataset Structure ### Data Instances ### Data Fields ### Data Splits

175000 lines of text each for train and test data. ## Dataset Creation ### Curation Rationale ### Source Data #### Initial Data Collection and Normalization #### Who are the source language producers? ### Annotations #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information ## Considerations for Using the Data ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations ## Additional Information ### Dataset Curators

The dataset was initially created by Thomas Martin ### Licensing Information

ODC Open Database License v1.0 ### Contributions

Thanks to @Shubhambindal2017 for adding this dataset.
TAGS
#task_categories-text-classification #annotations_creators-no-annotation #language_creators-found #multilinguality-multilingual #size_categories-100K<n<1M #source_datasets-original #language-Achinese #language-Afrikaans #language-Tosk Albanian #language-Amharic #language-Aragonese #language-Old English (ca. 450-1100) #language-Arabic #language-Egyptian Arabic #language-Assamese #language-Asturian #language-Avaric #language-Aymara #language-Azerbaijani #language-South Azerbaijani #language-Bashkir #language-Bavarian #language-Central Bikol #language-Belarusian #language-Bulgarian #language-Bhojpuri #language-Banjar #language-Bengali #language-Tibetan #language-Bishnupriya #language-Breton #language-Bosnian #language-Russia Buriat #language-Catalan #language-Chavacano #language-Min Dong Chinese #language-Chechen #language-Cebuano #language-Cherokee #language-Central Kurdish #language-Corsican #language-Crimean Tatar #language-Czech #language-Kashubian #language-Chuvash #language-Welsh #language-Danish #language-German #language-Dimli (individual language) #language-Lower Sorbian #language-Dotyali #language-Dhivehi #language-Emilian #language-Modern Greek (1453-) #language-English #language-Esperanto #language-Spanish #language-Estonian #language-Basque #language-Extremaduran #language-Persian #language-Finnish #language-Faroese #language-French #language-Arpitan #language-Friulian #language-Western Frisian #language-Irish #language-Gagauz #language-Scottish Gaelic #language-Galician #language-Gilaki #language-Guarani #language-Gujarati #language-Manx #language-Hausa #language-Hakka Chinese #language-Hebrew #language-Hindi #language-Fiji Hindi #language-Croatian #language-Upper Sorbian #language-Haitian #language-Hungarian #language-Armenian #language-Interlingua (International Auxiliary Language Association) #language-Indonesian #language-Interlingue #language-Igbo #language-Iloko #language-Ido #language-Icelandic #language-Italian #language-Japanese #language-Jamaican Creole English #language-Lojban #language-Javanese #language-Georgian #language-Kara-Kalpak #language-Kabyle #language-Kabardian #language-Kazakh #language-Khmer #language-Kannada #language-Korean #language-Komi-Permyak #language-Konkani (macrolanguage) #language-Karachay-Balkar #language-Kölsch #language-Kurdish #language-Komi #language-Cornish #language-Kirghiz #language-Latin #language-Ladino #language-Luxembourgish #language-Lezghian #language-Ganda #language-Limburgan #language-Ligurian #language-Lombard #language-Lingala #language-Lao #language-Northern Luri #language-Lithuanian #language-Latgalian #language-Latvian #language-Literary Chinese #language-Maithili #language-map #language-Moksha #language-Malagasy #language-Eastern Mari #language-Maori #language-Minangkabau #language-Macedonian #language-Malayalam #language-Mongolian #language-Marathi #language-Western Mari #language-Malay (macrolanguage) #language-Maltese #language-Mirandese #language-Burmese #language-Erzya #language-Mazanderani #language-Min Nan Chinese #language-Neapolitan #language-Norwegian Bokmål #language-Classical Nahuatl #language-Low German #language-Nepali (macrolanguage) #language-Newari #language-Dutch #language-Norwegian Nynorsk #language-Narom #language-Pedi #language-Navajo #language-Occitan (post 1500) #language-Livvi #language-Oromo #language-Oriya (macrolanguage) #language-Ossetian #language-Panjabi #language-Pangasinan #language-Pampanga #language-Papiamento #language-Picard #language-Pennsylvania German #language-Pfaelzisch #language-Polish #language-Western Panjabi #language-Pushto #language-Portuguese #language-Quechua #language-Romansh #language-Romanian #language-roa #language-Russian #language-Rusyn #language-Macedo-Romanian #language-Kinyarwanda #language-Sanskrit #language-Yakut #language-Sardinian #language-Sicilian #language-Scots #language-Sindhi #language-Samogitian #language-Serbo-Croatian #language-Sinhala #language-Slovak #language-Slovenian #language-Northern Sami #language-Shona #language-Somali #language-Albanian #language-Serbian #language-Sranan Tongo #language-Saterfriesisch #language-Sundanese #language-Swedish #language-Swahili (macrolanguage) #language-Silesian #language-Tamil #language-Tulu #language-Telugu #language-Tetum #language-Tajik #language-Thai #language-Turkmen #language-Tagalog #language-Tswana #language-Tonga (Tonga Islands) #language-Turkish #language-Tatar #language-Tuvinian #language-Udmurt #language-Uighur #language-Ukrainian #language-Urdu #language-Uzbek #language-Venetian #language-Veps #language-Vietnamese #language-Vlaams #language-Volapük #language-Võro #language-Walloon #language-Waray (Philippines) #language-Wolof #language-Wu Chinese #language-Xhosa #language-Mingrelian #language-Yiddish #language-Yoruba #language-Zeeuws #language-Chinese #license-odbl #language-identification #arxiv-1801.07779 #region-us 
 # Dataset Card for wili_2018 ## Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information
  - Contributions ## Dataset Description

- Homepage: URL
- Repository: 
- Paper: URL
- Leaderboard: 
- Point of Contact: Thoma, Martin (Email: info@URL) ### Dataset Summary

WiLI-2018, the Wikipedia language identification benchmark dataset, contains 235000 paragraphs of 235 languages. The dataset is balanced and a train-test split is provided. ### Supported Tasks and Leaderboards ### Languages

235 Different Languages ## Dataset Structure ### Data Instances ### Data Fields ### Data Splits

175000 lines of text each for train and test data. ## Dataset Creation ### Curation Rationale ### Source Data #### Initial Data Collection and Normalization #### Who are the source language producers? ### Annotations #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information ## Considerations for Using the Data ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations ## Additional Information ### Dataset Curators

The dataset was initially created by Thomas Martin ### Licensing Information

ODC Open Database License v1.0 ### Contributions

Thanks to @Shubhambindal2017 for adding this dataset.