license:apache-2.0 region:us
---
license: apache-2.0
---
Usage: (you need to specify **use_fast=False, trust_remote_code=True** as the args of *AutoTokenizer.from_pretrained()*)
```Python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "llm-jp/hf-slow-tokenizer-v21b1",
    legacy=True,
    use_fast=False,
    trust_remote_code=True,
)
```

When you create the tokenizer instance from the downloaded model with *trust_remote_code=True*, the model directory must be placed directly under the current directory, and the *pretrained_model_name_or_path* **should not start with "./"** but **should end with "/"**.
```Python
tokenizer = AutoTokenizer.from_pretrained(
    "hf-slow-tokenizer-v21b1/",
    legacy=True,
    use_fast=False,
    trust_remote_code=True,
)
```
