# japanese-reversed-gpt2-medium-unidic
This is a medium-sized Japanese **reversed** GPT-2 model using BERT-like tokenizer.
Unlike most Language Models, this model generates sentences from right to left.

Not reversed version is published [here](https://huggingface.co/okazaki-lab/japanese-gpt2-medium-unidic/).

# How to use
The model depends on [PyTorch](https://pytorch.org/), [fugashi](https://github.com/polm/fugashi) with [unidic-lite](https://github.com/polm/unidic-lite), and [Hugging Face Transformers](https://github.com/huggingface/transformers).

```sh
pip install torch torchvision torchaudio
pip install fugashi[unidic-lite]
pip install transformers
```

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
tokenizer = AutoTokenizer.from_pretrained('okazaki-lab/japanese-reversed-gpt2-medium-unidic')
model = AutoModelForCausalLM.from_pretrained('okazaki-lab/japanese-reversed-gpt2-medium-unidic')

text = 'ので、散歩に行きました。'

bos = tokenizer.convert_tokens_to_ids(['[BOS]']) # [32768]
input_ids = bos + tokenizer.encode(text)[1:-1][::-1] # [CLS] and [SEP] generated by BERT Tokenizer are removed then reversed
input_ids = torch.tensor(input_ids).unsqueeze(0)
output = model.generate(
    input_ids,
    do_sample=True,
    max_new_tokens=30,
    top_k=50,
    top_p=0.95,
    repetition_penalty=1.0,
    num_return_sequences=1,
    pad_token_id=0,
    eos_token_id=32769,
)[0].flip(0)

print(tokenizer.decode(output))
```

# Model architecture
Transformer-based Language Model
- Layers: 24
- Heads: 16
- Dimensions of hidden states: 1024

# Training
We used a [codebase](https://github.com/rinnakk/japanese-pretrained-models) provided by rinna Co., Ltd. for training.

The model was trained on Japanese CC-100 and Japanese Wikipedia (2022/01/31).
We employed 8 A100 GPUs for 17 days.
The perplexity on the validation set is 9.79.

# Tokenization
Our tokenizer is based on [the one](https://huggingface.co/cl-tohoku/bert-base-japanese-v2)  provided by Tohoku NLP Group.
The texts are tokenized by MeCab and then WordPiece.

The vocabulary size is 32771 (32768 original tokens + 2 special tokens + 1 unused token).

# License
[Creative Commons Attribution-ShareAlike 4.0](https://creativecommons.org/licenses/by-sa/4.0/)

Copyright (c) 2021, Tohoku University

Copyright (c) 2023, Tokyo Institute of Technology
{"language": ["ja"], "license": "cc-by-sa-4.0", "tags": ["gpt", "japanese", "language model", "reversed gpt-2"], "datasets": ["wikipedia", "cc100"], "pipeline_tag": "text-generation", "inference": false}
transformers pytorch gpt2 text-generation gpt japanese language model reversed gpt-2 ja dataset:wikipedia dataset:cc100 license:cc-by-sa-4.0 autotrain_compatible text-generation-inference region:us
TAGS
#transformers #pytorch #gpt2 #text-generation #gpt #japanese #language model #reversed gpt-2 #ja #dataset-wikipedia #dataset-cc100 #license-cc-by-sa-4.0 #autotrain_compatible #text-generation-inference #region-us 

# japanese-reversed-gpt2-medium-unidic
This is a medium-sized Japanese reversed GPT-2 model using BERT-like tokenizer.
Unlike most Language Models, this model generates sentences from right to left.

Not reversed version is published here.

# How to use
The model depends on PyTorch, fugashi with unidic-lite, and Hugging Face Transformers.





# Model architecture
Transformer-based Language Model
- Layers: 24
- Heads: 16
- Dimensions of hidden states: 1024

# Training
We used a codebase provided by rinna Co., Ltd. for training.

The model was trained on Japanese CC-100 and Japanese Wikipedia (2022/01/31).
We employed 8 A100 GPUs for 17 days.
The perplexity on the validation set is 9.79.

# Tokenization
Our tokenizer is based on the one  provided by Tohoku NLP Group.
The texts are tokenized by MeCab and then WordPiece.

The vocabulary size is 32771 (32768 original tokens + 2 special tokens + 1 unused token).

# License
Creative Commons Attribution-ShareAlike 4.0

Copyright (c) 2021, Tohoku University

Copyright (c) 2023, Tokyo Institute of Technology
# japanese-reversed-gpt2-medium-unidic
This is a medium-sized Japanese reversed GPT-2 model using BERT-like tokenizer.
Unlike most Language Models, this model generates sentences from right to left.

Not reversed version is published here. # How to use
The model depends on PyTorch, fugashi with unidic-lite, and Hugging Face Transformers. # Model architecture
Transformer-based Language Model
- Layers: 24
- Heads: 16
- Dimensions of hidden states: 1024 # Training
We used a codebase provided by rinna Co., Ltd. for training.

The model was trained on Japanese CC-100 and Japanese Wikipedia (2022/01/31).
We employed 8 A100 GPUs for 17 days.
The perplexity on the validation set is 9.79. # Tokenization
Our tokenizer is based on the one  provided by Tohoku NLP Group.
The texts are tokenized by MeCab and then WordPiece.

The vocabulary size is 32771 (32768 original tokens + 2 special tokens + 1 unused token). # License
Creative Commons Attribution-ShareAlike 4.0

Copyright (c) 2021, Tohoku University

Copyright (c) 2023, Tokyo Institute of Technology
TAGS
#transformers #pytorch #gpt2 #text-generation #gpt #japanese #language model #reversed gpt-2 #ja #dataset-wikipedia #dataset-cc100 #license-cc-by-sa-4.0 #autotrain_compatible #text-generation-inference #region-us 
 # japanese-reversed-gpt2-medium-unidic
This is a medium-sized Japanese reversed GPT-2 model using BERT-like tokenizer.
Unlike most Language Models, this model generates sentences from right to left.

Not reversed version is published here. # How to use
The model depends on PyTorch, fugashi with unidic-lite, and Hugging Face Transformers. # Model architecture
Transformer-based Language Model
- Layers: 24
- Heads: 16
- Dimensions of hidden states: 1024 # Training
We used a codebase provided by rinna Co., Ltd. for training.

The model was trained on Japanese CC-100 and Japanese Wikipedia (2022/01/31).
We employed 8 A100 GPUs for 17 days.
The perplexity on the validation set is 9.79. # Tokenization
Our tokenizer is based on the one  provided by Tohoku NLP Group.
The texts are tokenized by MeCab and then WordPiece.

The vocabulary size is 32771 (32768 original tokens + 2 special tokens + 1 unused token). # License
Creative Commons Attribution-ShareAlike 4.0

Copyright (c) 2021, Tohoku University

Copyright (c) 2023, Tokyo Institute of Technology
TAGS
#transformers #pytorch #gpt2 #text-generation #gpt #japanese #language model #reversed gpt-2 #ja #dataset-wikipedia #dataset-cc100 #license-cc-by-sa-4.0 #autotrain_compatible #text-generation-inference #region-us 
# japanese-reversed-gpt2-medium-unidic
This is a medium-sized Japanese reversed GPT-2 model using BERT-like tokenizer.
Unlike most Language Models, this model generates sentences from right to left.

Not reversed version is published here.# How to use
The model depends on PyTorch, fugashi with unidic-lite, and Hugging Face Transformers.# Model architecture
Transformer-based Language Model
- Layers: 24
- Heads: 16
- Dimensions of hidden states: 1024# Training
We used a codebase provided by rinna Co., Ltd. for training.

The model was trained on Japanese CC-100 and Japanese Wikipedia (2022/01/31).
We employed 8 A100 GPUs for 17 days.
The perplexity on the validation set is 9.79.# Tokenization
Our tokenizer is based on the one  provided by Tohoku NLP Group.
The texts are tokenized by MeCab and then WordPiece.

The vocabulary size is 32771 (32768 original tokens + 2 special tokens + 1 unused token).# License
Creative Commons Attribution-ShareAlike 4.0

Copyright (c) 2021, Tohoku University

Copyright (c) 2023, Tokyo Institute of Technology