
# Dataset Card for "tner/wikiann"

## Dataset Description

- **Repository:** [T-NER](https://github.com/asahi417/tner)
- **Paper:** [https://aclanthology.org/P17-1178/](https://aclanthology.org/P17-1178/)
- **Dataset:** WikiAnn
- **Domain:** Wikipedia
- **Number of Entity:** 3


### Dataset Summary
WikiAnn NER dataset formatted in a part of [TNER](https://github.com/asahi417/tner) project.
- Entity Types: `LOC`, `ORG`, `PER`

## Dataset Structure

### Data Instances
An example of `train` of `ja` looks as follows.

```
{
  'tokens': ['#', '#', 'ユ', 'リ', 'ウ', 'ス', '・', 'ベ', 'ー', 'リ', 'ッ', 'ク', '#', '1', '9','9','9'],
  'tags': [6, 6, 2, 5, 5, 5, 5, 5, 5, 5, 5, 5, 6, 6, 6, 6, 6]
}
```

### Label ID
The label2id dictionary can be found at [here](https://huggingface.co/datasets/tner/wikiann/raw/main/dataset/label.json).
```python
{
    "B-LOC": 0,
    "B-ORG": 1,
    "B-PER": 2,
    "I-LOC": 3,
    "I-ORG": 4,
    "I-PER": 5,
    "O": 6
}
```

### Data Splits

| language     |   train |   validation |   test |
|:-------------|--------:|-------------:|-------:|
| ace          |     100 |          100 |    100 |
| bg           |   20000 |        10000 |  10000 |
| da           |   20000 |        10000 |  10000 |
| fur          |     100 |          100 |    100 |
| ilo          |     100 |          100 |    100 |
| lij          |     100 |          100 |    100 |
| mzn          |     100 |          100 |    100 |
| qu           |     100 |          100 |    100 |
| su           |     100 |          100 |    100 |
| vi           |   20000 |        10000 |  10000 |
| af           |    5000 |         1000 |   1000 |
| bh           |     100 |          100 |    100 |
| de           |   20000 |        10000 |  10000 |
| fy           |    1000 |         1000 |   1000 |
| io           |     100 |          100 |    100 |
| lmo          |     100 |          100 |    100 |
| nap          |     100 |          100 |    100 |
| rm           |     100 |          100 |    100 |
| sv           |   20000 |        10000 |  10000 |
| vls          |     100 |          100 |    100 |
| als          |     100 |          100 |    100 |
| bn           |   10000 |         1000 |   1000 |
| diq          |     100 |          100 |    100 |
| ga           |    1000 |         1000 |   1000 |
| is           |    1000 |         1000 |   1000 |
| ln           |     100 |          100 |    100 |
| nds          |     100 |          100 |    100 |
| ro           |   20000 |        10000 |  10000 |
| sw           |    1000 |         1000 |   1000 |
| vo           |     100 |          100 |    100 |
| am           |     100 |          100 |    100 |
| bo           |     100 |          100 |    100 |
| dv           |     100 |          100 |    100 |
| gan          |     100 |          100 |    100 |
| it           |   20000 |        10000 |  10000 |
| lt           |   10000 |        10000 |  10000 |
| ne           |     100 |          100 |    100 |
| ru           |   20000 |        10000 |  10000 |
| szl          |     100 |          100 |    100 |
| wa           |     100 |          100 |    100 |
| an           |    1000 |         1000 |   1000 |
| br           |    1000 |         1000 |   1000 |
| el           |   20000 |        10000 |  10000 |
| gd           |     100 |          100 |    100 |
| ja           |   20000 |        10000 |  10000 |
| lv           |   10000 |        10000 |  10000 |
| nl           |   20000 |        10000 |  10000 |
| rw           |     100 |          100 |    100 |
| ta           |   15000 |         1000 |   1000 |
| war          |     100 |          100 |    100 |
| ang          |     100 |          100 |    100 |
| bs           |   15000 |         1000 |   1000 |
| eml          |     100 |          100 |    100 |
| gl           |   15000 |        10000 |  10000 |
| jbo          |     100 |          100 |    100 |
| map-bms      |     100 |          100 |    100 |
| nn           |   20000 |         1000 |   1000 |
| sa           |     100 |          100 |    100 |
| te           |    1000 |         1000 |   1000 |
| wuu          |     100 |          100 |    100 |
| ar           |   20000 |        10000 |  10000 |
| ca           |   20000 |        10000 |  10000 |
| en           |   20000 |        10000 |  10000 |
| gn           |     100 |          100 |    100 |
| jv           |     100 |          100 |    100 |
| mg           |     100 |          100 |    100 |
| no           |   20000 |        10000 |  10000 |
| sah          |     100 |          100 |    100 |
| tg           |     100 |          100 |    100 |
| xmf          |     100 |          100 |    100 |
| arc          |     100 |          100 |    100 |
| cbk-zam      |     100 |          100 |    100 |
| eo           |   15000 |        10000 |  10000 |
| gu           |     100 |          100 |    100 |
| ka           |   10000 |        10000 |  10000 |
| mhr          |     100 |          100 |    100 |
| nov          |     100 |          100 |    100 |
| scn          |     100 |          100 |    100 |
| th           |   20000 |        10000 |  10000 |
| yi           |     100 |          100 |    100 |
| arz          |     100 |          100 |    100 |
| cdo          |     100 |          100 |    100 |
| es           |   20000 |        10000 |  10000 |
| hak          |     100 |          100 |    100 |
| kk           |    1000 |         1000 |   1000 |
| mi           |     100 |          100 |    100 |
| oc           |     100 |          100 |    100 |
| sco          |     100 |          100 |    100 |
| tk           |     100 |          100 |    100 |
| yo           |     100 |          100 |    100 |
| as           |     100 |          100 |    100 |
| ce           |     100 |          100 |    100 |
| et           |   15000 |        10000 |  10000 |
| he           |   20000 |        10000 |  10000 |
| km           |     100 |          100 |    100 |
| min          |     100 |          100 |    100 |
| or           |     100 |          100 |    100 |
| sd           |     100 |          100 |    100 |
| tl           |   10000 |         1000 |   1000 |
| zea          |     100 |          100 |    100 |
| ast          |    1000 |         1000 |   1000 |
| ceb          |     100 |          100 |    100 |
| eu           |   10000 |        10000 |  10000 |
| hi           |    5000 |         1000 |   1000 |
| kn           |     100 |          100 |    100 |
| mk           |   10000 |         1000 |   1000 |
| os           |     100 |          100 |    100 |
| sh           |   20000 |        10000 |  10000 |
| tr           |   20000 |        10000 |  10000 |
| zh-classical |     100 |          100 |    100 |
| ay           |     100 |          100 |    100 |
| ckb          |    1000 |         1000 |   1000 |
| ext          |     100 |          100 |    100 |
| hr           |   20000 |        10000 |  10000 |
| ko           |   20000 |        10000 |  10000 |
| ml           |   10000 |         1000 |   1000 |
| pa           |     100 |          100 |    100 |
| si           |     100 |          100 |    100 |
| tt           |    1000 |         1000 |   1000 |
| zh-min-nan   |     100 |          100 |    100 |
| az           |   10000 |         1000 |   1000 |
| co           |     100 |          100 |    100 |
| fa           |   20000 |        10000 |  10000 |
| hsb          |     100 |          100 |    100 |
| ksh          |     100 |          100 |    100 |
| mn           |     100 |          100 |    100 |
| pdc          |     100 |          100 |    100 |
| simple       |   20000 |         1000 |   1000 |
| ug           |     100 |          100 |    100 |
| zh-yue       |   20000 |        10000 |  10000 |
| ba           |     100 |          100 |    100 |
| crh          |     100 |          100 |    100 |
| fi           |   20000 |        10000 |  10000 |
| hu           |   20000 |        10000 |  10000 |
| ku           |     100 |          100 |    100 |
| mr           |    5000 |         1000 |   1000 |
| pl           |   20000 |        10000 |  10000 |
| sk           |   20000 |        10000 |  10000 |
| uk           |   20000 |        10000 |  10000 |
| zh           |   20000 |        10000 |  10000 |
| bar          |     100 |          100 |    100 |
| cs           |   20000 |        10000 |  10000 |
| fiu-vro      |     100 |          100 |    100 |
| hy           |   15000 |         1000 |   1000 |
| ky           |     100 |          100 |    100 |
| ms           |   20000 |         1000 |   1000 |
| pms          |     100 |          100 |    100 |
| sl           |   15000 |        10000 |  10000 |
| ur           |   20000 |         1000 |   1000 |
| bat-smg      |     100 |          100 |    100 |
| csb          |     100 |          100 |    100 |
| fo           |     100 |          100 |    100 |
| ia           |     100 |          100 |    100 |
| la           |    5000 |         1000 |   1000 |
| mt           |     100 |          100 |    100 |
| pnb          |     100 |          100 |    100 |
| so           |     100 |          100 |    100 |
| uz           |    1000 |         1000 |   1000 |
| be-x-old     |    5000 |         1000 |   1000 |
| cv           |     100 |          100 |    100 |
| fr           |   20000 |        10000 |  10000 |
| id           |   20000 |        10000 |  10000 |
| lb           |    5000 |         1000 |   1000 |
| mwl          |     100 |          100 |    100 |
| ps           |     100 |          100 |    100 |
| sq           |    5000 |         1000 |   1000 |
| vec          |     100 |          100 |    100 |
| be           |   15000 |         1000 |   1000 |
| cy           |   10000 |         1000 |   1000 |
| frr          |     100 |          100 |    100 |
| ig           |     100 |          100 |    100 |
| li           |     100 |          100 |    100 |
| my           |     100 |          100 |    100 |
| pt           |   20000 |        10000 |  10000 |
| sr           |   20000 |        10000 |  10000 |
| vep          |     100 |          100 |    100 |

### Citation Information

```
@inproceedings{pan-etal-2017-cross,
    title = "Cross-lingual Name Tagging and Linking for 282 Languages",
    author = "Pan, Xiaoman  and
      Zhang, Boliang  and
      May, Jonathan  and
      Nothman, Joel  and
      Knight, Kevin  and
      Ji, Heng",
    booktitle = "Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2017",
    address = "Vancouver, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/P17-1178",
    doi = "10.18653/v1/P17-1178",
    pages = "1946--1958",
    abstract = "The ambitious goal of this work is to develop a cross-lingual name tagging and linking framework for 282 languages that exist in Wikipedia. Given a document in any of these languages, our framework is able to identify name mentions, assign a coarse-grained or fine-grained type to each mention, and link it to an English Knowledge Base (KB) if it is linkable. We achieve this goal by performing a series of new KB mining methods: generating {``}silver-standard{''} annotations by transferring annotations from English to other languages through cross-lingual links and KB properties, refining annotations through self-training and topic selection, deriving language-specific morphology features from anchor links, and mining word translation pairs from cross-lingual links. Both name tagging and linking results for 282 languages are promising on Wikipedia data and on-Wikipedia data.",
}
```
task_categories:token-classification task_ids:named-entity-recognition multilinguality:multilingual size_categories:10K<100k language:ace language:bg language:da language:fur language:ilo language:lij language:mzn language:qu language:su language:vi language:af language:bh language:de language:fy language:io language:lmo language:nap language:rm language:sv language:vls language:als language:bn language:diq language:ga language:is language:ln language:nds language:ro language:sw language:vo language:am language:bo language:dv language:gan language:it language:lt language:ne language:ru language:szl language:wa language:an language:br language:el language:gd language:ja language:lv language:nl language:rw language:ta language:war language:ang language:bs language:eml language:gl language:jbo language:nn language:sa language:te language:wuu language:ar language:ca language:en language:gn language:jv language:mg language:no language:sah language:tg language:xmf language:arc language:eo language:gu language:ka language:mhr language:nov language:scn language:th language:yi language:arz language:cdo language:es language:hak language:kk language:mi language:oc language:sco language:tk language:yo language:as language:ce language:et language:he language:km language:min language:or language:sd language:tl language:zea language:ast language:ceb language:eu language:hi language:kn language:mk language:os language:sh language:tr language:ay language:ckb language:ext language:hr language:ko language:ml language:pa language:si language:tt language:az language:co language:fa language:hsb language:ksh language:mn language:pdc language:ug language:ba language:crh language:fi language:hu language:ku language:mr language:pl language:sk language:uk language:zh language:bar language:cs language:hy language:ky language:ms language:pms language:sl language:ur language:csb language:fo language:ia language:la language:mt language:pnb language:so language:uz language:cv language:fr language:id language:lb language:mwl language:ps language:sq language:vec language:be language:cy language:frr language:ig language:li language:my language:pt language:sr region:us
{"language": ["ace", "bg", "da", "fur", "ilo", "lij", "mzn", "qu", "su", "vi", "af", "bh", "de", "fy", "io", "lmo", "nap", "rm", "sv", "vls", "als", "bn", "diq", "ga", "is", "ln", "nds", "ro", "sw", "vo", "am", "bo", "dv", "gan", "it", "lt", "ne", "ru", "szl", "wa", "an", "br", "el", "gd", "ja", "lv", "nl", "rw", "ta", "war", "ang", "bs", "eml", "gl", "jbo", "nn", "sa", "te", "wuu", "ar", "ca", "en", "gn", "jv", "mg", false, "sah", "tg", "xmf", "arc", "eo", "gu", "ka", "mhr", "nov", "scn", "th", "yi", "arz", "cdo", "es", "hak", "kk", "mi", "oc", "sco", "tk", "yo", "as", "ce", "et", "he", "km", "min", "or", "sd", "tl", "zea", "ast", "ceb", "eu", "hi", "kn", "mk", "os", "sh", "tr", "ay", "ckb", "ext", "hr", "ko", "ml", "pa", "si", "tt", "az", "co", "fa", "hsb", "ksh", "mn", "pdc", "ug", "ba", "crh", "fi", "hu", "ku", "mr", "pl", "sk", "uk", "zh", "bar", "cs", "hy", "ky", "ms", "pms", "sl", "ur", "csb", "fo", "ia", "la", "mt", "pnb", "so", "uz", "cv", "fr", "id", "lb", "mwl", "ps", "sq", "vec", "be", "cy", "frr", "ig", "li", "my", "pt", "sr"], "multilinguality": ["multilingual"], "size_categories": ["10K<100k"], "task_categories": ["token-classification"], "task_ids": ["named-entity-recognition"], "pretty_name": "WikiAnn"}
TAGS
#task_categories-token-classification #task_ids-named-entity-recognition #multilinguality-multilingual #size_categories-10K<100k #language-Achinese #language-Bulgarian #language-Danish #language-Friulian #language-Iloko #language-Ligurian #language-Mazanderani #language-Quechua #language-Sundanese #language-Vietnamese #language-Afrikaans #language-bh #language-German #language-Western Frisian #language-Ido #language-Lombard #language-Neapolitan #language-Romansh #language-Swedish #language-Vlaams #language-Tosk Albanian #language-Bengali #language-Dimli (individual language) #language-Irish #language-Icelandic #language-Lingala #language-Low German #language-Romanian #language-Swahili (macrolanguage) #language-Volapük #language-Amharic #language-Tibetan #language-Dhivehi #language-Gan Chinese #language-Italian #language-Lithuanian #language-Nepali (macrolanguage) #language-Russian #language-Silesian #language-Walloon #language-Aragonese #language-Breton #language-Modern Greek (1453-) #language-Scottish Gaelic #language-Japanese #language-Latvian #language-Dutch #language-Kinyarwanda #language-Tamil #language-Waray (Philippines) #language-Old English (ca. 450-1100) #language-Bosnian #language-Emiliano-Romagnolo #language-Galician #language-Lojban #language-Norwegian Nynorsk #language-Sanskrit #language-Telugu #language-Wu Chinese #language-Arabic #language-Catalan #language-English #language-Guarani #language-Javanese #language-Malagasy #language-Norwegian #language-Yakut #language-Tajik #language-Mingrelian #language-Official Aramaic (700-300 BCE) #language-Esperanto #language-Gujarati #language-Georgian #language-Eastern Mari #language-Novial #language-Sicilian #language-Thai #language-Yiddish #language-Egyptian Arabic #language-Min Dong Chinese #language-Spanish #language-Hakka Chinese #language-Kazakh #language-Maori #language-Occitan (post 1500) #language-Scots #language-Turkmen #language-Yoruba #language-Assamese #language-Chechen #language-Estonian #language-Hebrew #language-Khmer #language-Minangkabau #language-Oriya (macrolanguage) #language-Sindhi #language-Tagalog #language-Zeeuws #language-Asturian #language-Cebuano #language-Basque #language-Hindi #language-Kannada #language-Macedonian #language-Ossetian #language-Serbo-Croatian #language-Turkish #language-Aymara #language-Central Kurdish #language-Extremaduran #language-Croatian #language-Korean #language-Malayalam #language-Panjabi #language-Sinhala #language-Tatar #language-Azerbaijani #language-Corsican #language-Persian #language-Upper Sorbian #language-Kölsch #language-Mongolian #language-Pennsylvania German #language-Uighur #language-Bashkir #language-Crimean Tatar #language-Finnish #language-Hungarian #language-Kurdish #language-Marathi #language-Polish #language-Slovak #language-Ukrainian #language-Chinese #language-Bavarian #language-Czech #language-Armenian #language-Kirghiz #language-Malay (macrolanguage) #language-Piemontese #language-Slovenian #language-Urdu #language-Kashubian #language-Faroese #language-Interlingua (International Auxiliary Language Association) #language-Latin #language-Maltese #language-Western Panjabi #language-Somali #language-Uzbek #language-Chuvash #language-French #language-Indonesian #language-Luxembourgish #language-Mirandese #language-Pushto #language-Albanian #language-Venetian #language-Belarusian #language-Welsh #language-Northern Frisian #language-Igbo #language-Limburgan #language-Burmese #language-Portuguese #language-Serbian #region-us 

Dataset Card for "tner/wikiann"
===============================


Dataset Description
-------------------


* Repository: T-NER
* Paper: URL
* Dataset: WikiAnn
* Domain: Wikipedia
* Number of Entity: 3


### Dataset Summary


WikiAnn NER dataset formatted in a part of TNER project.


* Entity Types: 'LOC', 'ORG', 'PER'


Dataset Structure
-----------------


### Data Instances


An example of 'train' of 'ja' looks as follows.


### Label ID


The label2id dictionary can be found at here.


### Data Splits




### Dataset Summary


WikiAnn NER dataset formatted in a part of TNER project.


* Entity Types: 'LOC', 'ORG', 'PER'


Dataset Structure
----------------- ### Data Instances


An example of 'train' of 'ja' looks as follows. ### Label ID


The label2id dictionary can be found at here. ### Data Splits
TAGS
#task_categories-token-classification #task_ids-named-entity-recognition #multilinguality-multilingual #size_categories-10K<100k #language-Achinese #language-Bulgarian #language-Danish #language-Friulian #language-Iloko #language-Ligurian #language-Mazanderani #language-Quechua #language-Sundanese #language-Vietnamese #language-Afrikaans #language-bh #language-German #language-Western Frisian #language-Ido #language-Lombard #language-Neapolitan #language-Romansh #language-Swedish #language-Vlaams #language-Tosk Albanian #language-Bengali #language-Dimli (individual language) #language-Irish #language-Icelandic #language-Lingala #language-Low German #language-Romanian #language-Swahili (macrolanguage) #language-Volapük #language-Amharic #language-Tibetan #language-Dhivehi #language-Gan Chinese #language-Italian #language-Lithuanian #language-Nepali (macrolanguage) #language-Russian #language-Silesian #language-Walloon #language-Aragonese #language-Breton #language-Modern Greek (1453-) #language-Scottish Gaelic #language-Japanese #language-Latvian #language-Dutch #language-Kinyarwanda #language-Tamil #language-Waray (Philippines) #language-Old English (ca. 450-1100) #language-Bosnian #language-Emiliano-Romagnolo #language-Galician #language-Lojban #language-Norwegian Nynorsk #language-Sanskrit #language-Telugu #language-Wu Chinese #language-Arabic #language-Catalan #language-English #language-Guarani #language-Javanese #language-Malagasy #language-Norwegian #language-Yakut #language-Tajik #language-Mingrelian #language-Official Aramaic (700-300 BCE) #language-Esperanto #language-Gujarati #language-Georgian #language-Eastern Mari #language-Novial #language-Sicilian #language-Thai #language-Yiddish #language-Egyptian Arabic #language-Min Dong Chinese #language-Spanish #language-Hakka Chinese #language-Kazakh #language-Maori #language-Occitan (post 1500) #language-Scots #language-Turkmen #language-Yoruba #language-Assamese #language-Chechen #language-Estonian #language-Hebrew #language-Khmer #language-Minangkabau #language-Oriya (macrolanguage) #language-Sindhi #language-Tagalog #language-Zeeuws #language-Asturian #language-Cebuano #language-Basque #language-Hindi #language-Kannada #language-Macedonian #language-Ossetian #language-Serbo-Croatian #language-Turkish #language-Aymara #language-Central Kurdish #language-Extremaduran #language-Croatian #language-Korean #language-Malayalam #language-Panjabi #language-Sinhala #language-Tatar #language-Azerbaijani #language-Corsican #language-Persian #language-Upper Sorbian #language-Kölsch #language-Mongolian #language-Pennsylvania German #language-Uighur #language-Bashkir #language-Crimean Tatar #language-Finnish #language-Hungarian #language-Kurdish #language-Marathi #language-Polish #language-Slovak #language-Ukrainian #language-Chinese #language-Bavarian #language-Czech #language-Armenian #language-Kirghiz #language-Malay (macrolanguage) #language-Piemontese #language-Slovenian #language-Urdu #language-Kashubian #language-Faroese #language-Interlingua (International Auxiliary Language Association) #language-Latin #language-Maltese #language-Western Panjabi #language-Somali #language-Uzbek #language-Chuvash #language-French #language-Indonesian #language-Luxembourgish #language-Mirandese #language-Pushto #language-Albanian #language-Venetian #language-Belarusian #language-Welsh #language-Northern Frisian #language-Igbo #language-Limburgan #language-Burmese #language-Portuguese #language-Serbian #region-us 
 ### Dataset Summary


WikiAnn NER dataset formatted in a part of TNER project.


* Entity Types: 'LOC', 'ORG', 'PER'


Dataset Structure
----------------- ### Data Instances


An example of 'train' of 'ja' looks as follows. ### Label ID


The label2id dictionary can be found at here. ### Data Splits
passage: 