(feat Tokenizers): How to make models aware of structuring linebreaks?
Hi everyone!I’ll try to explain briefly the task I am trying to solve.Shortly:I would like my model to take into account newline markers in my text samples because I believe them to be highly informative in my case. Trying the methods that propose Transformers to insert new custom specials tokens yielded decreased performances.Long story :I have a bunch of official documents (IDs, driving licenses, etc…) from which I extract the textual content with OCR models. I want to see if I could train a unified NER-TokenClassification pipeline to parse different entities (lastname, firstname, birthdate, birthplace, etc…), that would generalize well to a wild variety of document kinds and countries.Methods based on regex patterns and/or computer vision work but can hardly be agnostic to the kind/language of the document and also suffer from OCR errors artefacts.For clarity sake, here’s what a basic sample coud look like : (fictive data)* * * PERMIS DE CONDUIRE RÉPUBLIQUE FRANÇAISE F *
1. DOE
2. JOHN
URQA 3.
01.01.1901 (UTOPIC CITY)
CONO
4a 01.02.2003
4b.01.02.2018
FRANCAIS
**I annotated my samples with the BIO-schema and trained a TokenClassification model basedgoogle/electra-large-discriminator. It worked like a charm, reaching high validation metrics!However, I noticed the model would sometime fail because\nnewlines markers are ignored during the tokenization and used as a splitting criteria.For instance is the example aboveJOHN\nURQA 3.would be splitted maybe as['JOHN', 'URQA', '3', '.'], making it ‘invisible’ to the model that there was actually a newline separator between the tokens.But that is precisely this newline indicator that makes it easy to realize that the actual entity isJOHNand that the OCR artefactURQAdoes not belong to the name.So I decided to preprocess my samples, replace\nmarkers by[NL]and adjust my BIO annotations accordingly (I don’t suspect a bug coming from this part).Then I loaded the-pretrained model to finetune as such:CHECKPOINT = 'google/electra-large-discriminator'

tokenizer = AutoTokenizer.from_pretrained(CHECKPOINT)
model = AutoModelForTokenClassification.from_pretrained(CHECKPOINT, num_labels=len(tag2id))

tokenizer.add_special_tokens({'additional_special_tokens': ['[NL]']})
model.resize_token_embeddings(len(tokenizer))
model.get_input_embeddings().padding_idx = 0Tokenization after this went as expected, not splitting the[NL]tokens and setting them a newtoken_id.Also the embedding matrix weights are unchanged afterresize_token_embeddings, except of course for a new line that has been inserted and initialized (for the special[NL]token).So I thought my model would now be aware of “newline splits” and behave even better. But actually the performances clearly degraded, which surprised me as newline breaks seem to be highly informative in this case.Here come the questions:Is this the correct way to handle custom token insertions ?By doing this, do I loose the pretrained weights/biases information, maybe explaining the reason performances went down ?What would be a good way for me to make the model aware of newlines markers ?Thank you for reading!(EDIT:the new[NL]special tokens are assigned the labelO, “no entity”.When I said performances decreased, more precisely the model was not able anymore to predict other things thatOlabel, ie. not detecting any entity. I will check if its able to overfit a small set of samples, haven’t tested this yet)
2022-04-12T16:50:21Z
Hi, did you ever figure out how to best address this? I’m in a similar situation with notes that have potentially informative line breaks…
2022-07-11T21:45:57Z
Great question! I would be very interested to hear the solution to this too
2023-03-07T06:25:34Z
@elliot-hActually the bert was trained with out line breaks ,unless you have a very large corpus of training data it wont be helpful for bert if you introduce a new character for line break Hope this helps
2023-08-24T00:07:53Z
I’m interested in this, too, for use withMarianMT with CTranslate2.