What is required to create a fast tokenizer? For example for a Marian model
I notice that there are no fast tokenizers available for the Opus Marian translation models. The library I’m using to run my models currently only supports fast tokenizers from HF, but they’re not available for Marian, is this due to a limitation unique to Marian, or would it be possible for me to create one? What would be involved in the process. I couldn’t find any information specific to creating a fast tokenizer from an existing slow tokenizer. Thanks!