transformers tokenizers transformers.js experimental auto tokenizer llama llama 2 custom code license:mit endpoints_compatible region:us
---
license: mit
tags:
- tokenizers
- transformers.js
- experimental
- auto tokenizer
- llama
- llama 2
- custom code
library_name: transformers
---

# Experimental tokenizer
- Derived from the LLaMa-2 tokenizer
- Inspired by the ChatML & Zephyr tokenizers