This repository doesn't contain a model, but only a tokenizer that can be used with the
tokenizers
library.
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_pretrained("anthony/tokenizers-test")
This repository doesn't contain a model, but only a tokenizer that can be used with the
tokenizers
library.
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_pretrained("anthony/tokenizers-test")