Feature to highlight or color code the text from the NER output of token classification having offsets using python
Feature requestI have fine tuned aHugging facetoken classificationmodel forNER task. I usepipelinefrom Hugging face to do prediction on test text data.I tag the data asBIOLformat.B stands of Beginning, I stand for Including, O means no entity, L means LastExample:Joh J Mathewwill be tagged asB_PERSONI_PERSONL_PERSONHere is how the output looks like:model = AutoModelForTokenClassification.from_pretrained("model_x")
tokenizer = AutoTokenizer.from_pretrained("model_x")
    
token_classifier = pipeline("token-classification", model=model, aggregation_strategy="max",tokenizer=tokenizer)

text=("""'IOWA DRIVER LICENSE 1 SAMPLE 2 MARK LIMITED-TERM 8 123 NORTH STREET APT 201 DES MOINES, IA 50301-1234 
Onom d DL No. 123XX6789 4a iss 1107/2016 4b exp 01/12/2021 15 Sex M 16 Hgt 5\'-08" 18 Eyes BRO 9a End NONE 9 
Class C 12 Rest NONE Mark Sample DONOR MED ALERT: Y HEARING IMP: Y MED ADV DIR: Y 3 OOB 01/12/1967 5 
DD 12345678901234567890123 NIVIA AL NA LANG ---- QUE EROL DE USA 01/12/67""")

for ent in token_classifier(text):
    print(ent)

{'entity_group': 'B_LAST_NAME', 'score': 0.9999994, 'word': 'SAMPLE', 'start': 23, 'end': 29}
{'entity_group': 'B_FIRST_NAME', 'score': 0.99999905, 'word': '', 'start': 32, 'end': 33}
{'entity_group': 'L_FIRST_NAME', 'score': 0.9999949, 'word': 'MARK', 'start': 32, 'end': 36}
{'entity_group': 'B_ADDRESS', 'score': 0.9999989, 'word': '123', 'start': 52, 'end': 55}
{'entity_group': 'I_ADDRESS', 'score': 0.99999917, 'word': 'NORTHSTREETAPT201DESMOINES,IA', 'start': 56, 'end': 91}
{'entity_group': 'I_DRIVER_LICENSE_NUMBER', 'score': 0.9999995, 'word': '123XX6789', 'start': 118, 'end': 127}
{'entity_group': 'L_ISSUE_DATE', 'score': 0.99999964, 'word': '1107/2016', 'start': 135, 'end': 144}
{'entity_group': 'I_EXPIRY_DATE', 'score': 0.99999964, 'word': '01/12/2021', 'start': 152, 'end': 162}
{'entity_group': 'B_PERSON_NAME', 'score': 0.99999905, 'word': 'Mark', 'start': 234, 'end': 238}
{'entity_group': 'I_PERSON_NAME', 'score': 0.9999993, 'word': 'Sample', 'start': 239, 'end': 245}
{'entity_group': 'L_DATE_OF_BIRTH', 'score': 0.99999976, 'word': '01/12/1967', 'start': 301, 'end': 311}So, given the offset valuesentity_group,word,start,endhow can I highlight the original text with theentity_groupso that it is easy to visulize.Final OutputIs there any python-library that I can use to do it.?MotivationMakes easy to visualise the NER outputYour contributionNA
2022-10-20T02:46:51Z
This looks like what you want.GitHub - natasha/ipymarkup: NER, syntax markup visualizationsimage1606×554 112 KB