Pegasus dropping Non-ASCII Chars
Hello all,I’m currently using a Pegasus model fine-tuned on a custom dataset to summarise conversations in English. Some participants in the conversation have non-ascii characters in their name e.g Gonçalo. I’ve noticed that for the original pegasus-large model and my fine tuned model these characters are dropped and the output references Gonalo not Gonçalo.This wasn’t the case with the BART model I was previously using.I was looking for help on how I could address this problem as I look to fine-tune my Pegasus model even further.Thanks,Karim
2022-03-03T13:56:55Z
Hi@kmfodawhich pegasus model are you using? In your custom dataset you say your input had special characters, but what about your output? In your tokenizer are you passing skip_special_tokens=True?
2022-03-03T14:01:43Z
Hi@anwarikathanks for your response. I’m using pegasus-large fine-tuned on my own dataset. I’m using the summarisation pipeline defaults - I’ll need to investigate what the default is for that variable.
2022-03-03T14:45:09Z
So it seems thatskip_special_tokensis set to True in the pipeline source code but there doesn’t seem to be a way pre-set the variable as False. It’s seems to be hardcoded in there if I’m not mistaken?Also, I’m not sure this would explain why if you feed Gonçalo into the hosted inference API example forBART-Largeyou get Gonçalo in the output whereas forPegasus-Largeyou get Gonalo. They both use the summarisation pipeline.
2022-03-04T14:03:50Z
The tokenizers are different, BART is aBPEand Pegasus usesSentencepiece. When you finetuned your pegasus model, did your data target data have those special characters? You may want to add certain non-ascii characters to your tokenizer and then fineTune it on your data.special_tokens_dict = {'additional_special_tokens': [['whatever your tokens translate to']]}
num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)
model.resize_token_embeddings(len(tokenizer))
2022-03-09T16:28:47Z
I see thanks so much. The target data did have those special characters yes. The code snipet you provided is very helpful I’ll try it out. Just wondering though, why is it added as a special token not a normal one?
2022-03-11T13:48:05Z
Recommend reading through this doc for your tokenizer type. Sorry not familiar with sentencepiece tokenizer.Summary of the tokenizersHere is a repo that discusses it in detailGitHub - google/sentencepiece: Unsupervised text tokenizer for Neural Network-based text generation.One part in that doc makes sense why your char don’t pass through.SentencePiece treats the input text just as a sequence of Unicode characters. Whitespace is also handled as a normal symbol. To handle the whitespace as a basic token explicitly, SentencePiece first escapes the whitespace with a meta symbol "▁" (U+2581) as follows.Reading further i think this doc may help you.