"OS Errorr: No space left on device" when trying to load a trained model from S3
Hello all!I have been stuck on this for weeks and am genuinely beyond confused. For some context, I was able to successfully train and finetune myCodeLlama-7BandCodeLlama-13Bon SageMaker using the instancesml.g5.2xlargeandml.g5.8xlargeand store these models in my S3 bucket. Then, I was able to effectively deploy myCodeLlama-7Bmodel on the SageMaker Inference Endpoint using the following code in my SageMaker Notebook Instance:model = HuggingFaceModel(
    model_data="s3://...model.tar.gz",
    entry_point="inference.py",
    source_dir="scripts",
    ... # some versioning parameters
)

predictor = model.deploy(
    endpoint_name="CodeLlama-7B",
    instance_type="ml.g5.2xlarge",
    ...
)In this code, themodel_datapoints to a file (model.tar.gz) containing my finetuned model andinference.pyis a script that holds the functions for inference (model_fn(),predict_fn(), etc.). Everything works beautifully when I deploy my CodeLlama-7B. However, when I replace it with my s3 file containing the CodeLlama-13B, I started receiving theOS Error: Device Out of Spaceerror. Several things I have tried thatall still resulted in this same error:Scaling up theinstance_typewith a very powerful instance, such asml.p4d.24xlarge(which is weird because I’ve seen tutorials hosting Llama 2-70B on this instance).Adding avolume_sizeparameter in mymodel.deploy()call with other large instances becauseml.g5.*instances don’t support attaching extra volume storage.Using multi-GPU and settingdevice_map='auto'when calling.from_pretrained().Setting theSM_NUM_GPUSvariable.Scale up my Notebook Instance.Any pointers and guidance would be very much appreciated!@philschmid, just wanted to say I’ve been following a lot of your tutorials and they have been super helpful, thank you so much for all the materials you’ve put out : )Cheers!
2023-12-28T00:27:20Z
The issue has been solved!My solution was to download the model on my local machine, unpack and include my custom inference script, and then call the.model()and .deploy()` without including the entry scripts and directories.