PEFT LoRA GPT-NeoX - LoraConfig
I have written a training script that makes use of the Accelerate and PEFT libraries to finetune GPT-NeoX and while the training loop runs without error the script does not generate an adapter_config.json file. My intuition is that there is something about my LoraConfig object that is not properly parameterized resulting in a silent failure.Does anyone have any suggestions on how to best parameterize LoraConfig for GPT-NEoX family of models?My current LoraConfig:peft_config = LoraConfig(
        r=16,
        lora_alpha=32,
        lora_dropout=0.05,
        target_modules = ["query_key_value", "xxx"],
        bias="none",
        task_type=TaskType.CAUSAL_LM,
        fan_in_fan_out=False,
    )My full training script can be foundhere.
2023-04-06T15:00:19Z
Hello@eusip!Thanks for the issue!Indeed you need to slightly tweak the trainer to add a callback to properly save your Peft models, please have a look at what have been suggested inIncorrect Saving Peft Models using HuggingFace Trainer · Issue #96 · huggingface/peft · GitHuband let us know if this works!
2023-04-06T16:08:26Z
@ybelkadaagain you have saved the day! Thanks for your help!