How to add additional custom pre-tokenization processing?
I would like to add a few custom functions for pre-tokenization. For example, I would like to split numerical text from any non-numerical test.Eg‘1000mg’ would become [‘1000’, ‘mg’].I am trying to figure out the proper way to do this for the python binding; I think it may be a bit tricky since its a binding for the original rust version.I am looking at the pretokenizer function/huggingface/tokenizers/blob/2ccd16bf5c3dd97759d7bdf5229e2feeba314b4a/bindings/python/py_src/tokenizers/pre_tokenizers/init.pyi#L6Which I am guessing may be where I could potentially as some pretokenization functions, but it doesn’t seem to return anything. I noticed that it’s expecting an instance of thePreTokenizedStringdefined here/huggingface/tokenizers/blob/2ccd16bf5c3dd97759d7bdf5229e2feeba314b4a/bindings/python/py_src/tokenizers/init.pyi#L55Which does seem to have some text processing functions. But they don’t seem to return anything. I am guessing that any additional rules need to be implemented in the original rust version itself?I am looking at the rust pretokenizers code, it seems that I have to add any additional preprocessing code heregithub.comhuggingface/tokenizers/blob/master/tokenizers/src/pre_tokenizers/unicode_scripts/pre_tokenizer.rsuse crate::pre_tokenizers::unicode_scripts::scripts::{get_script, Script};
use crate::tokenizer::{normalizer::Range, PreTokenizedString, PreTokenizer, Result};

#[derive(Clone, Debug)]
pub struct UnicodeScripts;
impl_serde_unit_struct!(UnicodeScriptsVisitor, UnicodeScripts);

impl UnicodeScripts {
    pub fn new() -> Self {
        Self {}
    }
}

impl Default for UnicodeScripts {
    fn default() -> Self {
        Self::new()
    }
}

// This code exists in the Unigram default IsValidSentencePiece.This file has been truncated.show originalDoes this seem like the right track for adding additional preprocessing code?It it makes a difference, what I am trying to do is train a brand new tokenizer.
2020-10-20T16:44:03Z
Hi@reSearch2vecThere are multiple ways to customize the pre-tokenization process:Using existing componentsThetokenizerslibrary provides many differentPreTokenizerthat you can use, and even combine as you wish to. There is a list of components in theofficial documentationUsing custom components written in PythonIt is possible to customize some of the components (Normalizer,PreTokenizer, andDecoder) using Python code. This hasn’t been documented yet, but you can find an examplehere. It lets you directly manipulate theNormalizedStringorPreTokenizedStringto normalize and pre-tokenize as you wish.Now for the example you mentioned (ie‘1000mg’ would become [‘1000’, ‘mg’]), you can probably use theDigitsPreTokenizerthat does exactly this.If you didn’t get a chance to familiarize yourself with theGetting startedpart of ourdocumentation, I think you will love it as it explains a bit more how to customize your tokenizer, and gives concrete examples.
2020-10-21T08:03:49Z
Thanks Anthony! A lot of great info.I didn’t know the tokenizers library had official documentation , it doesn’t seem to be listed on the github or pip pages, and googling ‘huggingface tokenizers documentation’ just gives links to the transformers library instead. It doesn’t seem to be on thehuggingface.comain page either.Very much looking forward to reading it.
2021-02-20T05:08:27Z
[quote=“anthony, post:2, topic:1637”]official documentation[/quote] points tolocalhost, could you update the link
2021-02-20T18:21:39Z
Thanks for letting me know! Just fixed it.
2021-03-30T15:13:38Z
Hi,I was able to create a Custom Pretokenizer based on the example linked above. But I’m not able to save the tokenizer due to the exception “Custom PreTokenizer cannot be serialized”. I’m wondering how to bypass this.
2023-03-07T04:40:09Z
Are there plans for this to become a documented part of the API? I notice that theCustomDecodercode no longer works (I believe the method name changed), it would be great to have a stable API for this stuff (although I get it’s a pretty niche thing)