
# marpa-tokenizer

> A LlamaTokenizer with support to tokenize Tibetan text.


Example:

```sh
['▁རྒྱལ་ཡོངས་', '▁ཀྱི་', '▁དོ་ཁུར་', '▁དང་', '▁འཛམ་གླིང་', '▁ཡོངས་', '▁ཀྱི་', '▁དོ་སྣང་', '▁ཁྲོད', '▁།']
['▁我们', '认为', '下面', '这些', '真理', '是不', '言', '而', '喻', '的']
```

{"language": ["bo"], "license": "mit", "tags": ["tibetan,tokenization,sentencepiece"]}
transformers pytorch llama feature-extraction tibetan,tokenization,sentencepiece bo license:mit endpoints_compatible text-generation-inference region:us
TAGS
#transformers #pytorch #llama #feature-extraction #tibetan,tokenization,sentencepiece #bo #license-mit #endpoints_compatible #text-generation-inference #region-us 


# marpa-tokenizer

> A LlamaTokenizer with support to tokenize Tibetan text.


Example:



# marpa-tokenizer

> A LlamaTokenizer with support to tokenize Tibetan text.


Example:
TAGS
#transformers #pytorch #llama #feature-extraction #tibetan,tokenization,sentencepiece #bo #license-mit #endpoints_compatible #text-generation-inference #region-us 
 # marpa-tokenizer

> A LlamaTokenizer with support to tokenize Tibetan text.


Example:
TAGS
#transformers #pytorch #llama #feature-extraction #tibetan,tokenization,sentencepiece #bo #license-mit #endpoints_compatible #text-generation-inference #region-us 
# marpa-tokenizer

> A LlamaTokenizer with support to tokenize Tibetan text.


Example: