GreetingsWould like to contribute to Amharic (less-resource language) I noticed resources for this language is next to nil.Where do I start and build up?1st datasets?2nd tokenizers ?3rd transformers?https://github.com/huggingface/tokenizers/issues/203#issue-586736250Thanks!
2021-02-19T14:14:01Z
Hello@yosiasz, welcome!Amharic is definitely an under-served language and one for which we’re working on improving coverage, but we do have some resources to get you started.First, you can look at the list of datasets that have some Amharic texthereFor example, you can get about 28M words of Amharic text from the CommonCrawl in theOSCAR datasetwith:import datasets

oscar_am = datasets.load_dataset("oscar", "unshuffled_deduplicated_am")
print(oscar_am["train"][0])We don’t have any models trained exclusively on Amharic so far, but we do have a few translation models as well as aLanguage-Agnostic Sentence Encoderyou can check out:import torch
from transformers import BertModel, BertTokenizerFast


tokenizer = BertTokenizerFast.from_pretrained("setu4993/LaBSE")
model = BertModel.from_pretrained("setu4993/LaBSE")
model = model.eval()

am_sentences = oscar_am["train"][:4]["text"]
am_inputs = tokenizer(am_sentences , truncation=True, return_tensors="pt", padding=True)

with torch.no_grad():
    am_outputs = model(**am_inputs )This will give you vector representations of all of the texts inam_sentencesHope that helps get you started, would you like to tell us more about what you want to do in Amharic so we can help further?
2021-02-19T17:12:37Z
@yjernitethanks very much! I want to fill in any missing gaps. You mention that “We don’t have any models trained exclusively on Amharic” I can see if I can contribute there?Also when I hear " Language-Agnostic Sentence Encoder" I feel nervous
2021-02-19T17:39:45Z
haha yup I don’t know what the downstream quality on Amharic specifically isSo one thing you could do that would be super useful would be to pre-train a BERT-style model for Amharic on OSCAR data, and maybe try fine-tuning it on theamportion of theWikiAnn NER datasetYou can find some instructions in theLanguage Modeling Tutorial(following the Masked Language Modeling approach)If you want to train your own tokenizer, you can: it may help you achieve better results. However, the MarianMT models already have a tokenizer that seems to work pretty well for English to Amharic translation - you can load it with:from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-am-sv", use_fast=True)Let us know how that goes and if you run into any trouble training your model!Also, you should definitely start an introduction thread for Amharic NLP in this forumYou can model it after the ones for Arabic and Korean if you need inspiration
2021-02-19T17:43:36Z
yjernite:import datasets

oscar_am = datasets.load_dataset("oscar", "unshuffled_deduplicated_am")
print(oscar_am["train"][0])Since Amharic is morpho complex language like other Semitic languages, Arabic work has been a great help in moving forward . will keep my eyes on themShukran!
2021-02-19T21:03:47Z
File "C:\PYTHON\3.6.8\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 58: character maps to <undefined>when runningimport datasets

oscar_am = datasets.load_dataset("oscar", "unshuffled_deduplicated_am")
print(oscar_am["train"][0])how to specify encoding UTF-8?
2021-02-19T21:56:44Z
Oh I haven’t seen that one before! Could you open an issue in the datasets library? That way we can tag all the relevant people. Is that the full trace or is there more?GitHubIssues · huggingface/datasets🤗 The largest hub of ready-to-use datasets for ML models with fast, easy-to-use and efficient data manipulation tools - Issues · huggingface/datasets
2021-02-19T21:57:51Z
looks like a windows issue.Create an issuegithub.com/huggingface/datasetsUnicodeDecodeError: windows 10 machineopened10:13PM - 19 Feb 21 UTCclosed10:40PM - 19 Feb 21 UTCyosiaszWindows 10
Php 3.6.8

when running

```
import datasets

oscar_am = data…sets.load_dataset("oscar", "unshuffled_deduplicated_am")
print(oscar_am["train"][0])
```
I get the following error

```
file "C:\PYTHON\3.6.8\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 58: character maps to <undefined>
```
2021-02-19T22:40:52Z
upgraded to php 3.9.2 and it works just fine.
2021-02-23T22:29:43Z
@yjernitea866×711 93.4 KBhad this problem while training Amharic BERT you suggested in this discussion.
2021-02-23T22:38:00Z
Hi@israel!I’d suggest posting in themain forumso we can help you debug this.There isn’t quite enough information in this screenshot, unfortunately: be sure to post the full notebook you’re running (e.g. as a Colab)!
2021-02-23T22:42:53Z
Amharic BERT TrainingBeginners@yjerniteProblem While training Amharic Language BERT on oscar dataset[a]Colab Link
2021-02-27T23:14:53Z
yjernite:WikiAnn NER dataseThis WikiAnn NER dataset does not have validation file (val.txt) Do I need that to do this#datasets = load_dataset("text", data_files={"train": 'data/amner/train'})
2021-02-27T23:38:23Z
all good nowtrainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=lm_datasets["train"],
        #eval_dataset=lm_datasets["validation"],
        eval_dataset=None,
        data_collator=data_collator,
    )