This is a toxic conversational model based on [tinkoff-ai/ruDialoGPT-medium](https://huggingface.co/tinkoff-ai/ruDialoGPT-medium). 

## Model training
We've created a custom dataset out of [raw imageboard dialogue data](https://github.com/Koziev/NLP_Datasets/tree/master/Conversations/Data).  
The data processing notebook is available [here](https://github.com/RakePants/nerdless/blob/main/notebooks/dataset.ipynb).

The model was finetuned on a chunk of the dataset of size 350,000 samples with the following parameters:
```
learning_rate=4e-7,
num_train_epochs=1, 
per_device_train_batch_size=24,
per_device_eval_batch_size=24,
warmup_steps=100,
gradient_accumulation_steps=16,
fp16=True
```

The finetuning notebook is available [here](https://github.com/RakePants/nerdless/blob/main/notebooks/finetuning.ipynb).
## Inference
You can utilize Better Transformers for faster inference.

The model can be inferenced as follows:
```
from optimum.bettertransformer import BetterTransformer
from transformers import AutoModelForCausalLM, AutoTokenizer

checkpoint = "rakepants/ruDialoGPT-medium-finetuned-toxic"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model_hf = AutoModelForCausalLM.from_pretrained(checkpoint)
model = BetterTransformer.transform(model_hf, keep_original_model=False)

# token id 50257 - @@ПЕРВЫЙ@@
# token id 50258 - @@ВТОРОЙ@@

input = "@@ПЕРВЫЙ@@Привет, как дела?@@ВТОРОЙ@@"  
inputs = tokenizer(input, return_tensors='pt')

generated_token_ids = model.generate(
    **inputs,
    top_k=10,
    top_p=0.95,
    num_beams=3,
    num_return_sequences=1,
    do_sample=True,
    no_repeat_ngram_size=2,
    temperature=0.7,
    repetition_penalty=1.2,
    length_penalty=1.0,
    early_stopping=True,
    max_new_tokens=48,
    eos_token_id=50257,
    pad_token_id=0
)

context_with_response = [tokenizer.decode(sample_token_ids) for sample_token_ids in generated_token_ids]
```
{"language": ["ru"], "license": "mit", "library_name": "transformers", "tags": ["gpt2", "conversational", "not-for-all-audiences"], "base_model": "tinkoff-ai/ruDialoGPT-medium", "inference": true, "widget": [{"text": "@@\u041f\u0415\u0420\u0412\u042b\u0419@@\u041f\u0440\u0438\u0432\u0435\u0442, \u043a\u0430\u043a \u0434\u0435\u043b\u0430?@@\u0412\u0422\u041e\u0420\u041e\u0419@@", "example_title": "Greet"}, {"text": "@@\u041f\u0415\u0420\u0412\u042b\u0419@@\u0422\u044b \u043d\u043e\u0440\u043c\u0430\u043b\u044c\u043d\u044b\u0439 \u0432\u043e\u043e\u0431\u0449\u0435?@@\u0412\u0422\u041e\u0420\u041e\u0419@@", "example_title": "Confront"}]}
transformers safetensors gpt2 text-generation conversational not-for-all-audiences ru base_model:tinkoff-ai/ruDialoGPT-medium license:mit autotrain_compatible endpoints_compatible text-generation-inference region:us