NER on multiple languages
I want to do NER task on news articles that are in dozens of languages. Is the best option to go forxlm-roberta-large-finetuned-conll03-english? I read that XLM models fine-tuned for a language work well in other languages as well. My main issue is that this model is too big. Should I go for language specific smaller models if I already know what language I’m dealing with?Also I’m curious, why does thexlm-roberta-large-finetuned-conll03-germanhave so much more downloads than the English one?
2020-08-06T11:36:46Z
Hi@goutham794,you could train a multi-lingual NER model on the WikiANN dataset (or better: use the train/dev/test partitioned fromGitHub - afshinrahimi/mmner: Massively Multilingual Transfer for NER).But fine-tuning one big multi-lingual NER model could be very complicated (fine-tuning instabilities). And you should keep in mind, that WikiANN only has three label types.If you already know what languages you want to cover, then a better way would be to train “mono-lingual” models + just search for NER datasets for your desired languages. Good resource is:GitHubGitHub - juand-r/entity-recognition-datasets: A collection of corpora for...A collection of corpora for named entity recognition (NER) and entity recognition tasks. These annotated datasets cover a variety of languages, domains and entity types. - GitHub - juand-r/entity-r...