Loading a locally saved model is very slow
Hello all, I am loading a llama 2 model from  HF hub on g5.48xlarge on sagemaker notebook instance using the below commands and it take less than 5 minutes to completes the loadingmodel_id = "meta-llama/Llama-2-7b-chat-hf"
tokenizer = AutoTokenizer.from_pretrained(model_id)
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(model_id,load_in_8bit=True, torch_dtype=torch.float16)Thereafter I save the model locally using the below codesave_directory = "models/Llama-2-7b-chat-hf"
model.save_pretrained(save_directory)
tokenizer.save_pretrained(save_directory)then I try loading the locally saved model using the below codemodel = AutoModelForCausalLM.from_pretrained(save_directory, device_map = 'auto')
tokenizer = AutoTokenizer.from_pretrained(save_directory)but it took around 30 minutes to load the same model from local as compared to less than 5 minute from HF hub, what can be the reason behind it and how can I load the model faster.Any answer would be appreciated, Thanks.
2024-07-10T14:53:25Z
Hi there, did you ever find a solution for this? - having the same issues here. Have run this code:config = AutoConfig.from_pretrained(storage_model_path)tokenizer = AutoTokenizer.from_pretrained(storage_model_path)model = AutoModelForCausalLM.from_pretrained(storage_model_path,config=config,torch_dtype=torch.bfloat16,trust_remote_code=True,local_files_only=True,)And it still takes 30 mins as opposed to 45 seconds when loading from the hub directly.Env requirements:transformers==4.41.2torch==2.2.2requests==2.31.0accelerate==0.31.0Using Databricks 14.3 ML cluster with cuda version 11.8 - not sure if its a read bits per second setting or on the transformers side?Would appreciate if anyone has a fix for this?