tokenizers-test / README.md
anthony's picture
anthony HF staff
Add readme
e8c6cb9 unverified
|
raw
history blame
223 Bytes

This repository doesn't contain a model, but only a tokenizer that can be used with the tokenizers library.

from tokenizers import Tokenizer
tokenizer = Tokenizer.from_pretrained("anthony/tokenizers-test")