PEFT version
have been trying to fine-tune the llama using alpaca-lora repo, i am able to train it without any error.
The issue is after training the adapter_model.bin is just 443 bytes, here are the commands that i used for training
python finetune.py --base_model 'decapoda-research/llama-7b-hf' --data_path './alpaca_data_2k.json' --output_dir './lora-alpaca-2k-6h' --val_set_size 256 --num_epochs=10 --cutoff_len=512 --group_by_length --lora_target_modules='[q_proj,k_proj,v_proj,o_proj]' --lora_r=16 --micro_batch_size=8
explanation:
- so this command is to train it in slurm based cluster (my uni. cluster). and i have generic script that accepts the pyenv and pyscript to execute after activating that env.
- also the data i have sampled 2k examples from alpaca.json
- i tried to scroll through finetune.py but couldn't get what's missing.
would really appriciate if anyone can point out the missing argument or command.
link to my hf model: 0mij/llama-v1-alpaca-2k
the only difference i saw is peft version.
Well, I just wonder how you could find this repo? I mean, I just create it minutes ago...
For your problem: firstly, the parameters shown in the command line do not match the parameters in the adapter_config.json, so does the adapter_model.bin in 0mij/llama-v1-alpaca-2k is actually created by the command you mentioned above?
thanks for the reply, yes i forgot to mention that the model i pushed to hf, was trained using default command, which does have these 2 modules as target. the command i shared was new one, because like you i also thought that this might be problem. i have downgraded peft version so it's in training, will get back here if it resolves.
thanks!
regarding your curiosity about finding this repo. i looked into all the lora weights for llama model on hf, but they all were old, so i searched using filter and you just pushed the weights, i compared with your adaperconfig.json and just created discussion, i also saw someone else with peft0.5dev was having only 443bytes .bin file similar to me.
After some digging, I find it a known issue on https://github.com/tloen/alpaca-lora. To be specifical, tloen save the state_dict of the peft model in a tricky way which may not work in the latest PEFT:
ref: https://github.com/tloen/alpaca-lora/blob/630d1146c8b5a968f5bf4f02f50f153a0c9d449d/finetune.py#L256-L261
old_state_dict = model.state_dict
model.state_dict = (
lambda self, *_, **__: get_peft_model_state_dict(
self, old_state_dict()
)
).__get__(model, type(model))
Please ref https://github.com/huggingface/peft/issues/286#issuecomment-1501617281 for more details.
By the way, I implement the Lora code on my own instead of using tloen's and I am 100% certain that the code above can be replaced by
model.save_pretrained("./llama_lora_alpaca")
Of course, downgrading PEFT can solve the problem as well.
Thanks a lot for your comment. I read the comments from the issue above, and its not the problem with peft, so downgrading didn't worked.
Now, i will test by commenting this code block, and will see.
it worked! thanks 😊