How to ensure the `overflow` with `stride` always starts with a full word?
In continuation tomy previous question,When I tokenize a sentence tomax_lengthwhile keeping the overflowing tokens which are returned with astride(overlap) on the previous segment, as is in the code below:tokenizer= SomeTokenizer.from_pretrained('some/path')
tokenizer('I am Nasheed and I like xylophones.', truncation=True, max_length=12, return_overflowing_tokens=True, stride=7)The above snippet segments my original string as below:Segment1: ‘I am Nasheed and I like xylo’Segment2: ‘heed and I like xylophones.’I want to ensure that the overflow segments I get always start with whole words as below*:Segment1: ‘I am Nasheed and I like’Segment2: ‘Nasheed and I like’Segment3: ‘and I like xylophones.’InSegment1I have removed the partial wordxylo, this can be done using what has been suggestedhere