license:apache-2.0 region:us
---
license: apache-2.0
---
The Hugging Face fast tokenizer for LLM-jp 13B.

The vocab size is 50,570.

Requirements:
- transformers>=4.34.0
- tokenizers>=0.14.0
- torch

Usage:
```Python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("llm-jp/hf-fast-tokenizer-v21b3")
```