Llama2 fine-tunning with PEFT QLora and testing the model
hi All,@philschmid, I hope you are doing well. Sorry for fine tuning llama2, I create csv file with the Alpaca structure which has text column including ### instruction ### input ### response, for fine tuning the model I am confused which method with PEFT and QLora should I use, I am confused with many codes, would you please refer me to any code that is right for fine tuning with alpaca structure, and saving and inference for testing the model? In some code I saw they did tokenizer truncate and padding and refer label to -100 and in other no preprocessing is done. I appreciate your help. Many thanks.
2023-08-13T23:49:34Z
I suggest using llama recipes repo from meta.
2023-08-14T08:56:24Z
I’d recommend checking out the official example scripts:TRL:https://github.com/huggingface/trl/blob/main/examples/scripts/sft_trainer.py. This script illustrates how to do supervised fine-tuning (SFT) on a dataset of (human instruction, completion) pairs. It can optionally leverage LoRa instead of full fine-tuning in case you pass--use_peftto the script. Seethis postfor an example on how to call the script. Do note that it will train your model on both instructions and completions by default (as it usesDataCollatorForLanguageModelingby default). You can alternatively use theDataCollatorForCompletionOnlyLM classto train on completions only. If you pass in both --load_in_4bit and --use_peft, then you’re doing QLoRa (quantized LoRa, i.e. LoRa on a frozen quantized LLM).PEFT itself also has example scripts and notebooks:https://github.com/huggingface/peft/tree/main/examples/causal_language_modeling.
2023-08-17T21:14:12Z
@nielsrall the examples seem to load a model into a single GPU before fine-tuning it. I found that using Deepspeed only the training is done in parallel, but the initial model needs to first fit to the single GPU. Any examples you are aware of that show how to fine tune a model that is loaded across GPUs? Thanks.
2023-08-18T06:27:32Z
@nielsr, many thanks for your help and tips. sorry, I can’t load my data in hugging face can I read it directly in the code from local CSV file? including one column names “text” which is concatenation of ### Human: . ### Assistant?   for inference is ther available code ?
2023-08-21T07:31:34Z
@aytugkaya, have you used multiple GPUs? is it possible to share your code with me if it is multiple GPUS?
2023-08-21T11:25:34Z
No, single Gpu only.
2023-08-21T12:21:20Z
Hello! I initiated a model and tokenization to train on a T4 GPU using://model_id=“meta-llama/Llama-2-7b-hf”tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id,load_in_8bit=True, device_map=‘auto’, torch_dtype=torch.float16)//The configuration I used://lora_config=LoraConfig(task_type=TaskType.CAUSAL_LM,inference_mode=False,r=8,lora_alpha=32,lora_dropout=0.05,target_modules = [“q_proj”, “v_proj”])config = {‘lora_config’: lora_config,‘learning_rate’: 1e-4,‘num_train_epochs’: 1,‘gradient_accumulation_steps’: 4,‘per_device_train_batch_size’: 1}training_args = TrainingArguments(output_dir=output_dir,overwrite_output_dir=True,fp16=True,  # Use BF16 if available# logging strategieslogging_dir=f"{output_dir}/logs",logging_strategy=“steps”,logging_steps=10,save_strategy=“no”,optim=“adamw_torch_fused”,max_steps=total_steps if enable_profiler else -1,**{k:v for k,v in config.items() if k != ‘lora_config’})model.save_pretrained(“/content/drive/MyDrive/Colab Notebooks/llama2/saved_model”)//To load the trained model and use it, I did://model = LlamaForCausalLM.from_pretrained(model_id,return_dict=True,load_in_8bit=True,device_map=“auto”,low_cpu_mem_usage=True,)model = PeftModel.from_pretrained(model, peft_model,is_trainable=True, torch_dtype=torch.float16)//The inference is working and i don´t want to load checkpoint.When I try to do new fine-tuning with the trained model, I encounter this error://OutOfMemoryError: CUDA out of memory. Tried to allocate 256.00 MiB (GPU 0; 14.75 GiB total capacity; 13.34 GiB already allocated; 128.81 MiB free; 13.56 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF//How can i do it?
2023-08-21T13:04:17Z
I’ve got it now!The first time I fine-tuned, I did://model = prepare_model_for_int8_training(model)model = get_peft_model(model, peft_config)//When I loaded the model to continue training:://peft_model=“/content/drive/MyDrive/Colab Notebooks/llama2/saved_model2”model_id=“meta-llama/Llama-2-7b-hf”tokenizer = AutoTokenizer.from_pretrained(model_id)model = LlamaForCausalLM.from_pretrained(model_id,return_dict=True,load_in_8bit=True,device_map=“auto”,low_cpu_mem_usage=True,)model = prepare_model_for_int8_training(model) # <<<<---- I forgot this partmodel = PeftModel.from_pretrained(model, peft_model,is_trainable=True, torch_dtype=torch.float16)//
2023-08-22T08:26:00Z
hello,@dametodataare you using multiple gpus? or one. I am aftering a code which work with multiple defined gpus
2023-08-22T10:51:56Z
Hello! I used one!
2023-12-04T01:25:33Z
Hi, i’m following the sft.py example to fine tune themeta-llama/Llama-2-7b-chat-hfwith this datasetmlabonne/guanaco-llama2-1k · Datasets at Hugging Face. i turned on load_in_4bits and perf and fine tuned the model for 30 epochs. the loss showing in the end has reached 0.05 ish. However, when I load this saved model and do inference, I always got same results as the vanilla one. I don’t know what’s the issue cus the training looks alright. Can someone shed some lights on this issue? thanks
2023-12-21T10:05:42Z
Hi,An update here is that we now have scripts for both supervised fine-tuning (SFT) and DPO (direct preference optimization) in the Alignment Notebook repository:alignment-handbook/scripts at main · huggingface/alignment-handbook · GitHub.These scripts support multi-GPU fine-tuning using DeepSpeed.
2023-12-21T10:06:45Z
Veryyyyy niceee thankss