How to disable Huggingface Hub during Trainer saving of PEFT models?
Hello everyone!I hope you are all doing well.I am currently training some LoRAs (via PEFT) using the Transformers Trainer. The Huggingface Hub was down this evening for less than an hour, but that caused my training to crash. This despite the fact that I amnotpushing my model to the Hub and I am saving my checkpoints locally.Please bear with me while I try to explain the chain of events:Whenever the trainer is saving a checkpoint it callsTrainer._save().Within this function there is amodel.save_pretrained()call.Because I’m using a peft model, this call corresponds to the function inPeftModel.There, we can find a call toget_peft_model_state_dict()frompeft.utils.This call in turns includes a call tofile_exists()fromhuggingface_hubSince the hub was down while the checkpoint was trying to be saved (locally) I got the following error:requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out.My question is: Is there a way of disabling this constant calling to the Huggingface_Hub during theTrainersaving phase? Perhaps a way to make thefile_exists()check on a cached copy of the model. In theory my training loop should be able to be run completely offline once the model was originally downloaded, but this dependency on checking the hub is making the process very dependent on a stable connection for what seems to me like a very simple file check.I did not post this as an Issue because I think it’s probably just a configuration fail on my configuration of the trainer rather than a shortcoming of the libraries.I’ll tag@muellerzrsince he is listed as a contact forTrainerIssues on the Transformers github.Thank you in advance for any help.
2024-04-23T17:22:09Z
You can probably disable it during training, iirc it’sTRANSFORMERS_OFFLINE=1. However there should be an option for turning off the REDACTED_TOKEN upload in theTrainingArguments
2024-04-24T13:13:26Z
Hello and thank you for the quick answer! I will give this a try.