Handling tokenization effects of punctuated numbers in NER (e.g. $10,000)
I have previously successfully fine-tuned an NER/Token classification model on a custom dataset (roughly based on this tutorial) - however in attempting to shift to a new dataset I have come across an issue.The new feature of the dataset, is that it contains numbers/dollar values e.g. of the form $123,000/1,000, $1,000,000 etc. The model is failing to classify these in full -and it’s made me realize I am probably fundamentally misunderstanding something about the sub-tokenization process - and was hoping someone could shed some light for me.Here is an example few rows in my datasetimage884×216 35.9 KBYou will note that I have already split my sentences/documents into words - this was done simply on white space FYI - so generating my training data I am usingis_split_into_wordsin the tokenizer.The model trains well - 0.87 F1 score on theB-asking_pricetag I am highlighting here.However after I export the code as a pipeline, and try to run the full sentence (i.e. not broken up into words) through it in a different environment, I get the following result:image1682×332 71.5 KBI am showing one example here, but this is pretty much consistent across all numbers I test it on. A couple of my thoughts:It’s splits the number into sub-tokens passed on punctuation (,$.etc.) - but doesn’t aggregate it again post-prediction. In the training data generation process I passed in$4,299,000as a token (understanding that the tokenizer would further split it up into sub-tokens). Is there a way to similarly pass word by word into a predictionpipelineso that it understands where to aggregate?It’s pretty consistent in that it will predictB-<LABEL>for the first token andB-<LABEL>orI-<LABEL>for the next tokens up until the first comma, after which it will predict theOcategory with high confidence.My uniformed guess is that the$4,and the299,000are getting split up in the pre-processing step of training because of the punctuation, with the latter being assigned no label? I feel like the solution would be to enforce aggregation at the known word e.g. “$4,299,000”, and usingaggregation_strategy=firstit would get assigned the correct label - but I’m not sure if there is an OOTB way to do this?Posting this here, because I feel like I missing a trick with how this sub-tokenization works & I’m sure someone has had to deal with punctuation in numbers before. Any ideas welcome
2022-02-11T04:16:11Z
So I dove in pretty deep & made many edits to thepipelineto work out the core issue & get something that worked. Afterwards I took a step-back and wrote something easier that simply wraps the output of calls topipeline(see below)Notes:Setaggregation_strategy=Nonein the pipelinexis the output of the huggingface token classification pipeline‘desired_tokens’ is the list of tokens that you want to aggregate to, e.g. in my example above `[“Price:”, " $4,290,000", “Number”…]I have learnt a-lot by diving in - but I have been left confused whyis_split_into_wordsin the tokenizer doesn’t behave like I thought it would. Even though I explicitly pass$4,290,000as it’s own word, the tokenizer still splits it up into it’s components - but without the##that would classify them as sub-tokens to be re-aggregated.def aggregate(x, desired_tokens):

        
        joined_word = ''
        joined_group = []
        
        new_x = []
            
        for i in x:
            joined_word += i['word'].replace('#','')
            joined_group.append(i)
            if joined_word == desired_tokens[0]:
                new_i = {'entity':joined_group[0]['entity'], 'label':joined_group[0]['label'], 'word':joined_word, 
                         'start':joined_group[0]['start'], 'end':joined_group[-1]['end'], 'score':joined_group[0]['score']}
                new_x.append(new_i)
        
                joined_word = ''
                joined_group = []
                desired_tokens = desired_tokens[1:]
        return new_x
2023-03-30T15:38:20Z
Hi,I have the same issue, I don’t understand how to solve it. Someone found an answer?Thanks