transformers safetensors mistral text-generation generated_from_trainer conversational base_model:mistralai/Mistral-7B-v0.1 license:apache-2.0 autotrain_compatible endpoints_compatible text-generation-inference region:us
---
license: apache-2.0
base_model: mistralai/Mistral-7B-v0.1
tags:
- generated_from_trainer
model-index:
- name: mist-7b-sft-gutenberg-50k
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# mist-7b-sft-gutenberg-50k

This model is a fine-tuned version of <br>
[mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) <br>
on a part of the train1 split of  
[sanps/GutenbergFictionSummaryPrepared](https://huggingface.co/datasets/sanps/GutenbergFictionSummaryPrepared)

It achieves the following results on the evaluation set (split: "small_val"):
- Loss: 2.1665

Loss curves:

![image/png](https://cdn-uploads.huggingface.co/production/uploads/62e271e0003eac16191c1724/Xz4CI5DuV32oi6xgA-fQ1.png)

I used the [https://github.com/huggingface/alignment-handbook](https://github.com/huggingface/alignment-handbook) with few minor changes.

Weirdly enough it stopped training at 0.69 epoch (no, I am not making this up!). Will have to debug why.

The training was kind of unstable in the beginning but I lowered the lr and added max_grad_norm. However I feel the lr is too low and could have got better results with higher lr.

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 2
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- num_devices: 2
- total_train_batch_size: 4
- total_eval_batch_size: 8
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- num_epochs: 1

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 2.1619        | 0.69  | 4282 | 2.1664          |


### Framework versions

- Transformers 4.35.0
- Pytorch 2.1.0+cu118
- Datasets 2.14.6
- Tokenizers 0.14.1
