Add readme
Browse files
README.md
ADDED
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
This repository doesn't contain a model, but only a tokenizer that can be used with the
|
2 |
+
`tokenizers` library.
|
3 |
+
|
4 |
+
```python
|
5 |
+
from tokenizers import Tokenizer
|
6 |
+
tokenizer = Tokenizer.from_pretrained("anthony/tokenizers-test")
|
7 |
+
```
|