tahaenesaslanturk commited on
Commit
06b1138
·
verified ·
1 Parent(s): 6cdb8d0

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +3 -3
README.md CHANGED
@@ -1,7 +1,7 @@
1
  ---
2
  license: mit
3
  ---
4
- # TS-Corpus BPE Tokenizer (32k, Uncased)
5
 
6
  ## Overview
7
  This repository hosts a Byte Pair Encoding (BPE) tokenizer with a vocabulary size of 32,000, trained uncased using several datasets from the TS Corpus website. The BPE method is particularly effective for languages like Turkish, providing a balance between word-level and character-level tokenization.
@@ -18,11 +18,11 @@ The tokenizer was trained on a variety of text sources from TS Corpus, ensuring
18
  The inclusion of idiomatic expressions, proverbs, and legal terminology provides a comprehensive toolkit for processing Turkish text across different domains.
19
 
20
  ## Tokenizer Model
21
- Utilizing the Byte Pair Encoding (BPE) method, this tokenizer excels in efficiently managing subword units without the need for an extensive vocabulary. BPE is especially suitable for handling the agglutinative nature of Turkish, where words can have multiple suffixes. This uncased version normalizes input text by converting all characters to lowercase, which simplifies processing and improves consistency.
22
 
23
  ## Usage
24
  To use this tokenizer in your projects, load it with the Hugging Face `transformers` library:
25
  ```python
26
  from transformers import AutoTokenizer
27
- tokenizer = AutoTokenizer.from_pretrained("tahaenesaslanturk/ts-corpus-bpe-32k-uncased")
28
  ```
 
1
  ---
2
  license: mit
3
  ---
4
+ # TS-Corpus BPE Tokenizer (32k, Cased)
5
 
6
  ## Overview
7
  This repository hosts a Byte Pair Encoding (BPE) tokenizer with a vocabulary size of 32,000, trained uncased using several datasets from the TS Corpus website. The BPE method is particularly effective for languages like Turkish, providing a balance between word-level and character-level tokenization.
 
18
  The inclusion of idiomatic expressions, proverbs, and legal terminology provides a comprehensive toolkit for processing Turkish text across different domains.
19
 
20
  ## Tokenizer Model
21
+ Utilizing the Byte Pair Encoding (BPE) method, this tokenizer excels in efficiently managing subword units without the need for an extensive vocabulary. BPE is especially suitable for handling the agglutinative nature of Turkish, where words can have multiple suffixes.
22
 
23
  ## Usage
24
  To use this tokenizer in your projects, load it with the Hugging Face `transformers` library:
25
  ```python
26
  from transformers import AutoTokenizer
27
+ tokenizer = AutoTokenizer.from_pretrained("tahaenesaslanturk/ts-corpus-bpe-32k-cased")
28
  ```