peft region:us
---
library_name: peft
---
## Model
"bigscience/bloomz-560m"

## Dataset
"Abirate/english_quotes" (English quotes)

## Training procedure
PEFT using LoRA 

lora configuration:
    - r=1,
    - lora_alpha=3,
    - target_modules=["query_key_value"],
    - lora_dropout=0.05, 
    - bias="none",
    - task_type="CAUSAL_LM"

trainable params: 98,304 || all params: 559,312,896 || trainable%: 0.01757585078102687
None

training with Pytorch using the folowing parameters:
    - learning_rate= 1e-2
    - num_train_epochs=10

### Framework versions

- PEFT 0.4.0