peft llama not-for-all-audiences license:agpl-3.0 4-bit bitsandbytes region:us
---
license: agpl-3.0
library_name: peft
tags:
- not-for-all-audiences
---

# Mistral-Pippa-7b-qlora

This is a repository of my Mistral-7b Qlora checkpoints of the [PIPPA-ShareGPT](https://huggingface.co/datasets/kingbri/PIPPA-ShareGPT) dataset.

You can read more about the dataset on its relevant page. It's a ShareGPT reformat of the [PIPPA dataset](https://huggingface.co/datasets/PygmalionAI/PIPPA) by PygmalionAI. The reformat was done to allow for axolotl compatability.

### Architecture

- **Model Architecture**: Mistral-7B
- **Training Algorithm**: QLora
- **Dataset Used**: PIPPA-ShareGPT (pippa_sharegpt_trimmed.jsonl)

### Training Details

- **Dataset**: [PIPPA-ShareGPT](https://huggingface.co/datasets/kingbri/PIPPA-ShareGPT)
- **Datset type**: ShareGPT
- **Training Parameters**: [See Here](https://pastebin.com/sUfbn6zr)
- **Training Environment**: Axolotl
- **sequence_len**: 4096

## Instruct Format

ShareGPT gets converted to vicuna format. The dataset uses modified roles of `USER` and `CHARACTER` instead of `USER` and `ASSISTANT`.

```
SYSTEM: Enter roleplay mode...
USER: {prompt}
CHARACTER:
```

## Notes

This Qlora was produced as an experiment to see how the public version of PIPPA can affect a model. Also, Mistral is fairly new and training/finetune can be broken. As a result, I have no idea if this lora is of great quality or absolute garbage.

## Acknowledgments

Thanks to:
- PygmalionAI: The creators of the PIPPA dataset
- Axolotl: Finetuning suite
- [Kingbri](https://huggingface.co/kingbri): The OG author of this LoRA who helped me a lot

## Donate?
If you'd like to donate to Kingbri, you can do so here: [https://ko-fi.com/kingbri](https://ko-fi.com/kingbri)

If you'd like to donate to me, you can also do it here: [https://ko-fi.com/undiai](https://ko-fi.com/undiai)

You should not feel obligated to donate, but if you do, we'll appreciate it.

## Axolotl stuff

## Training procedure


The following `bitsandbytes` quantization config was used during training:
- quant_method: bitsandbytes
- load_in_8bit: False
- load_in_4bit: True
- llm_int8_threshold: 6.0
- llm_int8_skip_modules: None
- llm_int8_enable_fp32_cpu_offload: False
- llm_int8_has_fp16_weight: False
- bnb_4bit_quant_type: nf4
- bnb_4bit_use_double_quant: True
- bnb_4bit_compute_dtype: bfloat16

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 16
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 10
- num_epochs: 3

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 1.6025        | 0.27  | 50   | 1.6366          |
| 1.5383        | 0.53  | 100  | 1.5604          |
| 1.5321        | 0.8   | 150  | 1.5271          |
| 1.4347        | 1.07  | 200  | 1.5094          |
| 1.4273        | 1.34  | 250  | 1.5019          |
| 1.4772        | 1.6   | 300  | 1.4944          |
| 1.4244        | 1.87  | 350  | 1.4879          |
| 1.3786        | 2.14  | 400  | 1.4922          |
| 1.3493        | 2.41  | 450  | 1.4917          |
| 1.3949        | 2.67  | 500  | 1.4918          |
| 1.3663        | 2.94  | 550  | 1.4917          |


### Framework versions

- Transformers 4.34.0.dev0
- Pytorch 2.0.1+cu118
- Datasets 2.14.5
- Tokenizers 0.14.0
