peft llama2 license:other region:us
---
license: other
library_name: peft
tags:
- llama2
---


# Model Card for PIPPA ShareGPT Subset Variation Two Lora 7b

It is an experimental Lora focused on Roleplay that uses a subset of [PIPPA ShareGPT](https://huggingface.co/datasets/kingbri/PIPPA-shareGPT), the difference from the previous variant is that it was trained with different parameters micro_batch_size = 1 and gradient_accumulation_steps = 1
## Usage

Custom
```
SYSTEM: Do thing
USER: {prompt}
CHARACTER:
```

## Bias, Risks, and Limitations

This Lora is not intended for supplying factual information or advice in any form

## Training Details

### Training Data

1k of conversation from [PIPPA ShareGPT](https://huggingface.co/datasets/kingbri/PIPPA-shareGPT)

### Training Procedure 

The version of this Lora uploaded on this repository was trained using a 8x RTX A6000 cluster in 8-bit with regular LoRA adapters and 32-bit AdamW optimizer.

#### Training Hyperparameters

Training using a fork of [Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) with two paths [Patch 1](https://github.com/bdashore3/axolotl/commit/995557bdf3c6c8b3e839b224ef9513fc2b097f30) [Patch 2](https://github.com/bdashore3/axolotl/commit/8970280de2ea01e41c044406051922715f4086cb)

- load_in_8bit: true
- lora_r: 16
- lora_alpha: 16
- lora_dropout: 0.01
- gradient_accumulation_steps: 1
- micro_batch_size: 1
- num_epochs: 3
- learning_rate: 0.000065

## Environmental Impact

Finetuning this model on 4xNVIDIA A6000 48GB in parallel takes about 45 minutes (7B)