|
--- |
|
language: |
|
- ja |
|
- zh |
|
- ko |
|
license: cc-by-sa-4.0 |
|
datasets: |
|
- wikipedia |
|
mask_token: "[MASK]" |
|
widget: |
|
- text: "早稲田大学で自然言語処理を[MASK]ぶ。" |
|
- text: "李白是[MASK]朝人。" |
|
- text: "불고기[MASK] 먹겠습니다." |
|
--- |
|
|
|
### Model description |
|
- This model was trained on **ZH, JA, KO**'s Wikipedia (5 epochs). |
|
|
|
### How to use |
|
```python |
|
from transformers import AutoTokenizer, AutoModelForMaskedLM |
|
tokenizer = AutoTokenizer.from_pretrained("conan1024hao/cjkbert-small") |
|
model = AutoModelForMaskedLM.from_pretrained("conan1024hao/cjkbert-small") |
|
``` |
|
- Before you fine-tune downstream tasks, you don't need any text segmentation. |
|
- (Though you may obtain better results if you applied morphological analysis to the data before fine-tuning) |
|
|
|
### Morphological analysis tools |
|
- ZH: For Chinese, we use [LTP](https://github.com/HIT-SCIR/ltp). |
|
- JA: For Japanese, we use [Juman++](https://github.com/ku-nlp/jumanpp). |
|
- KO: For Korean, we use [KoNLPy](https://github.com/konlpy/konlpy)(Kkma class). |
|
|
|
### Tokenization |
|
- We use character-based tokenization with **whole-word-masking** strategy. |
|
|
|
### Model size |
|
- vocab_size: 15015 |
|
- num_hidden_layers: 4 |
|
- hidden_size: 512 |
|
- num_attention_heads: 8 |
|
- param_num: 25M |
