Generate tokenizer.json for Marian(Opus) MT
Hi all,I am deploying MT model on an Android device with Java. The Deep Java Lib (DJL) support the Autotokenizer of Huggingface, but it only supports the one with the tokenizer.json file. However, the tokenizer of Opus MT requires SnetencePiece files including source.spm, target.spm and vocab.json. Is there any way to convert the files to one tokenizer.json so I can apply it in DJL?
2024-02-26T13:04:02Z
@hasanybody solved this problem?