German Sentiment (tokenizer)?
Hi,I want to use German Sentiment (" German Sentiment Classification with Bert") repository (Link:GitHub - oliverguhr/german-sentiment-lib: An easy to use python package for deep learning-based german sentiment classification.).I have a dataset of tweets in German language. Do I need to use tokenizer before applying this model? And if yes which one?In the repository as an example no tokenizer is being used.Thank you!
2023-02-10T12:36:06Z
Hi@Al3ksandra,the repo uses theoliverguhr/german-sentiment-bertmodel:github.comoliverguhr/german-sentiment-lib/blob/master/germansentiment/sentimentmodel.py#L7from transformers import AutoModelForSequenceClassification, AutoTokenizerfrom typing import Listimport torchimport reclass SentimentModel():def __init__(self, model_name: str = "oliverguhr/german-sentiment-bert"):if torch.cuda.is_available():self.device = 'cuda'else:self.device = 'cpu'self.model = AutoModelForSequenceClassification.from_pretrained(model_name)self.model = self.model.to(self.device)self.tokenizer = AutoTokenizer.from_pretrained(model_name)self.clean_chars = re.compile(r'[^A-Za-züöäÖÜÄß ]', re.MULTILINE)Which is basically a fine-tuned model of thebert-base-german-casedon sentiment datasets. This can be verified when comparing the vocab of the original German BERT model and the fine-tuned one:$ curl https://huggingface.co/oliverguhr/german-sentiment-bert/resolve/main/vocab.txt > vocab_sentiment.txt
$ curl https://huggingface.co/bert-base-german-cased/resolve/main/vocab.txt > vocab_german_bert.txt

$ diff vocab_sentiment.txt vocab_german_bert.txtIt has no difference.You do not need to use your own tokenizer/cleanup steps, because it has built-in methods. So when calling thepredict_sentimentmethod, your input text will be tokenized and cleaned before:github.comoliverguhr/german-sentiment-lib/blob/367f8f55d92fd85e6cde8bc59dc8dbad7ec88071/germansentiment/sentimentmodel.py#L25self.tokenizer = AutoTokenizer.from_pretrained(model_name)self.clean_chars = re.compile(r'[^A-Za-züöäÖÜÄß ]', re.MULTILINE)self.clean_http_urls = re.compile(r'https*\S+', re.MULTILINE)self.clean_at_mentions = re.compile(r'@\S+', re.MULTILINE)def predict_sentiment(self, texts: List[str], output_probabilities = False)-> List[str]:texts = [self.clean_text(text) for text in texts]# Add special tokens takes care of adding [CLS], [SEP], <s>... tokens in the right way for each model.# truncation=True limits number of tokens to model's limitations (512)encoded = self.tokenizer.batch_encode_plus(texts,padding=True, add_special_tokens=True,truncation=True, return_tensors="pt")encoded = encoded.to(self.device)with torch.no_grad():logits = self.model(**encoded)label_ids = torch.argmax(logits[0], axis=1)if output_probabilities == False:return [self.model.config.id2label[label_id.item()] for label_id in label_ids]else:predictions = torch.softmax(logits[0], dim=-1).tolist()I hope this helps
2023-02-10T15:29:24Z
Thanks Stefan! It helps