Tokenizing using JS
I’ve exported a custom PyTorch-based Transformer model into ONNX to run it on NodeJS. However, the exported model seems to expectinput_idsdirectly (and not raw text).Is there any way I can perform tokenization in JS?Or is there something I’m missing, wherein the ONNX model itself is capable of performing the tokenization as well?
2022-08-17T06:05:49Z
I have the same problem. Seems converting to onnx is only half the battle. Maybe I’ll write a library. How hard could it be?
2022-08-19T09:46:08Z
I have written a JavaScript library that is capable of running the T5 tokenizer:transformers-js/tokenizers.js at main · praeclarum/transformers-js · GitHub
2023-02-22T21:39:50Z
Awesome library@praeclarum! I used it as inspiration when developinghttps://github.com/xenova/transformers.js, which supports BERT, DistilBERT, T5 and GPT2 tokenization and inference.
2023-05-05T17:07:24Z
Thank you, so so much. This library finally made it possible for even a noob such as myself to execute my models in the browser. I’d probably get a C in my Data Mining class if I didn’t find this amazing tool.