tokenizer en license:llama2 region:us
---
license: llama2
language:
- en
tags:
- tokenizer
---


A copy of Meta's [llama](meta-llama/Llama-2-7b) tokenizer, with three tokens added to mask PII:

- `|||EMAIL_ADDRESS|||`
- `|||PHONE_NUMBER|||`
- `|||IP_ADDRESS|||`