Generating NER data sample for RoBERTa model
I have a trained RoBERTa model with a Byte Level BPE Encoding algorithm, which I want to benchmark on a custom NER dataset.Each sample is as followed:Text: John is playing footballLabels: B-PER O O OThe text could be run through the tokenizer to generate subword tokens. However, the total count of the subwords tokens may be different from the text tokens, and I don’t know how to align the labels accordingly
2020-09-06T08:16:06Z
cc@stefan-it
2020-09-14T01:53:56Z
Hello, it’s this topiv still relevant?Vào 15:26, CN, 6 Th9, 2020 Suraj Patil via Hugging Face Forums <hellohellohello@discoursemail.com> đã viết:
2020-09-14T02:11:16Z
Maybe something in here can help you:https://github.com/huggingface/transformers/tree/master/examples/token-classification