Downloading and storing models
Following thisblogpostI download the OPT175B model usingmodel = AutoModelForCausalLM.from_pretrained("bigscience/bloom", device_map="balanced_low_0", torch_dtype=torch.float16,cache_dir=<path to driec>)I can see a 350GB file (it took quite sometime to download but that is okay) created after in the<path to direct>.However, upon restarting the session, two behaviors are observedmodel = AutoModelForCausalLM.from_pretrained("bigscience/bloom", device_map="balanced_low_0", torch_dtype=torch.float16,cache_dir=<path to driec>)quickly loads the model (i.e., does not redownload the 350gb file) since it is found in<path to driec>. This is the desirable behaviorHowever, tryinggenerator = pipeline("text-generation", model="bigscience/bloom", cache_dir=<path to driec>,device_map="balanced_low_0", torch_dtype=torch.float16)begins redownaloding the model and I am not sure why.Moreover the files in<path to direc>haveblobs, refs, snapshotsin it but not a.jsonfile which probably what thepipeline()in searching for.My questions arehow to make pipeline() use the downloaded model instead of re-downloadingwhy isn’t from_pretrained()downloading the .json file?One way I read was to usemodel.save_pretrained(<path>)and then usepipeline(<path>)to load but this takes an extra 350gb of space sincemodel.save_pretrained(<path>)will create a new model with.jsonfile in it.Any suggestions?
2022-10-18T20:40:24Z
Oooh I think I figured it out.I think the correct way to usepipeline()is to do something like the following -model = AutoModelForCausalLM.from_pretrained("bigscience/bloom", device_map="balanced_low_0", torch_dtype=torch.float16,cache_dir=<path to driec>)followed bytoken = AutoTokenizer.from_pretrained("bigscience/bloom", cache_dir="/home/racball/opt175b_tokeniser")Then,generator = pipeline("text-generation", model=model,tokenizer=token)