arxiv:2305.14718 region:us

## HH-RLHF QLoRA adapters trained with Advantage Leftover Lunch RL with KL penalty (A-LoL KL)
| [Paper](https://arxiv.org/abs/2305.14718) | [Code](https://github.com/abaheti95/LoL-RL) |

### Model description
We continued QLoRA finetuning on [HH-RLHF dataset](https://huggingface.co/datasets/Anthropic/hh-rlhf) starting from [timdettmers/qlora-hh-rlhf-7b](https://huggingface.co/timdettmers/qlora-hh-rlhf-7b) for 1 epoch while only the "chosen" responses and removing the "rejected" responses from the training split. Even within the chosen responses, our method Advantage Leftover Lunch RL (A-LoL KL), inherently finds 33% of the responses as negative advantage and thus discards them as unfit for training. Despite the low number of training examples, the final adapter trained with A-LoL KL is able to generate diverse, safe and helpful responses compared to the baselines.

⚠️ These models are purely intended for research purposes and could produce problematic outputs.


### Usage

```python
tokenizer = AutoTokenizer.from_pretrained("huggyllama/llama-7b")
tokenizer.truncation_side = "left"
tokenizer.padding_side = "left"
base_model = AutoModelForCausalLM.from_pretrained(
    "huggyllama/llama-7b",
    load_in_4bit=True,
    torch_dtype=torch.float16,
    device_map="auto",
    max_memory= {i: '40000MB' for i in range(torch.cuda.device_count())},
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_compute_dtype=torch.float16,
        bnb_4bit_quant_type='nf4'
    ),
)
model = PeftModel.from_pretrained(base_model, "abaheti95/a_lol_kl_good_prioirty_qlora_hh")

prefix = "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.\n### Human: How would I go about making homemade chewing gum? ### Assistant: Well, it\u2019s pretty easy, but you can either make it a bit better or more interesting if you use xanthan gum, which is a commonly used thickener. Xanthan gum is added to food products as a thickener, gelling agent, emulsifier, and stabilizer, which means that it allows sugar syrups to form gels. If you can get hold of some, you could use it to make really interesting gum! ### Human: Okay what do I do when I have the xanthan gum? What else do I need to add? ### Assistant: Well, the important ingredient is the gums.  You can find many different types at your local pharmacy, which should be able to help you figure out what you need.  You will need:  1) a mixer, 2) sugar, 3) xanthan gum, 4) flavoring, 5) gum base, and 6) a dehydrator, which is a special oven that dries food. ### Human: Why would I need a dehydrator to make gum? ### Assistant:"

current_batch_prefixes_inputs = tokenizer([prefix], max_length = 768 - 128,truncation = True,add_special_tokens=True, padding = True, return_tensors="pt").to(model.device)
# 835 = "###"
predicted_sents = model.generate(**current_batch_prefixes_inputs, max_new_tokens=128,pad_token_id=tokenizer.pad_token_id,num_beams=1,do_sample=False,num_return_sequences = 1, eos_token_id=[835, tokenizer.eos_token_id])
repsonse_tokens = predicted_sents[:, current_batch_prefixes_inputs['input_ids'].shape[-1]:]
responses = tokenizer.batch_decode(repsonse_tokens, skip_special_tokens=True)
# Normalize responses
responses_normalized = [resp.split("\n Human:")[0].split("\nHuman:")[0].split("\n### Human")[0].strip() for resp in responses]
responses_normalized = [resp.replace("###", "").strip() if resp.endswith("###") else resp.strip() for resp in responses_normalized]
```

We also show the evaluation results of the model on the test set in file: `harmless_base_eval_results.jsonl`, `helpful_base_eval_results.jsonl`, `helpful_online_eval_results.jsonl` and `helpful_rejection_eval_results.jsonl`.


### Framework version and configuration
- PEFT 0.5.0

The following `bitsandbytes` quantization config was used during training:
- quant_method: bitsandbytes
- load_in_8bit: False
- load_in_4bit: True
- llm_int8_threshold: 6.0
- llm_int8_skip_modules: None
- llm_int8_enable_fp32_cpu_offload: False
- llm_int8_has_fp16_weight: False
- bnb_4bit_quant_type: nf4
- bnb_4bit_use_double_quant: False
- bnb_4bit_compute_dtype: float16

