RoBERTa Index out of range error on relation extraction data
Hi, I’m fine-tuningRoberta-largefor Relation Classification Task. It works fine on datasets havingTACREDformat. For this specific purpose, I have extracted text from pdf files using OCR tools. The extracted text containsalphanumerics,merged tokens, andwrongly spelledwords. Means to say, these tokens don’t seem to be originally present in English Vocabulary due to the OCR tool that extracts everything from PDF files.Whenever I run training, it throws Index out of range error as shown in the image attached.Do Roberta correctly tokenize these types of words mentioned.Also is there any way in the implementations to handle new words/tokens that are not in the lookup table of model vocabulary?Screenshot from 2022-04-29 00-08-161920×1080 220 KBWhat might be the possible solution so that we don’t have to care what word we are feeding to the model and run smoothly.NOTE: I prepared conll data from the same text for fine-tuning bert and it works smothly on similar tokens.