peft region:us
---
library_name: peft
---
## Training procedure


The following `bitsandbytes` quantization config was used during training:
- load_in_8bit: True
- load_in_4bit: False
- llm_int8_threshold: 6.0
- llm_int8_skip_modules: None
- llm_int8_enable_fp32_cpu_offload: False
- llm_int8_has_fp16_weight: False
- bnb_4bit_quant_type: fp4
- bnb_4bit_use_double_quant: False
- bnb_4bit_compute_dtype: float32
### Framework versions


- PEFT 0.4.0

- Here is the code to load the load along with llama2 chat model:

!pip install -q accelerate==0.21.0 peft==0.4.0 bitsandbytes==0.40.2 transformers==4.31.0 trl==0.4.7

import os, torch, logging
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, HfArgumentParser, TrainingArguments, pipeline
from peft import LoraConfig, PeftModel
from trl import SFTTrainer
import torch
from peft import PeftModel, PeftConfig
from transformers import AutoModelForCausalLM, AutoTokenizer

config = PeftConfig.from_pretrained(path_to_directory_containing adapter_config.json)
model = AutoModelForCausalLM.from_pretrained("NousResearch/Llama-2-7b-chat-hf", return_dict=True, load_in_8bit=True, device_map='auto')
tokenizer = AutoTokenizer.from_pretrained("NousResearch/Llama-2-7b-chat-hf")

model = PeftModel.from_pretrained(model, "/content/llama-2-7b-medichat")
batch = tokenizer("My friend has been feeling a little dizzy these days.What could he pissibely be suffering from ?", return_tensors='pt')
batch = batch.to('cuda')
output_tokens = model.generate(**batch, max_new_tokens=300)  ### You can adjust the max_new_tokens
output_response = tokenizer.decode(output_tokens[0], skip_special_tokens=True)