Sagemaker endpoint - no space left on device with large models
I’m trying to launch a sagemaker endpoint using one of the larger pretrained models but i kept getting out of disk errors. I found this very odd since i was using a machine that came with multiple terrabytes of diskdef model_fn(model_dir):

  logging.set_verbosity_info()
  logger = logging.get_logger("model_fn")
  result = subprocess.run(['df', '-kh'], stdout=subprocess.PIPE)
  logger.info(result)gave me the output:Filesystem      Size  Used Avail Use% Mounted on
overlay          52G   31G   22G  59% /
tmpfs            64M     0   64M   0% /dev
tmpfs            94G     0   94G   0% /sys/fs/cgroup
shm              92G   20K   92G   1% /dev/shm
/dev/nvme1n1    3.5T  196K  3.3T   1% /tmp
/dev/nvme0n1p1   52G   31G   22G  59% /etc/hosts
tmpfs            94G   12K   94G   1% /proc/driver/nvidia
devtmpfs         94G     0   94G   0% /dev/nvidia0
tmpfs            94G     0   94G   0% /proc/acpi
tmpfs            94G     0   94G   0% /sys/firmwareThis basically told me that everything was going to the overlay disk and that the disk that had all the space was assigned /tmp.So how do i make my overlay disk larger or how do i make it so that the model directory use /tmp instead of the default?My logs tell mecom.amazonaws.ml.mms.wlm.WorkerThread - Connecting to: /home/model-server/tmp/.mms.sock.9000Which i’m going to guess is where the model will attempt to download to?Anyone know the best way to deal with this?  Thanks!
2023-02-21T08:06:13Z
Hello@blackknight467,Thank you for researching this. I ll forward this to the SageMaker Team!In the meantime, what i can suggest is that you deploy your model using amodel.tar.gz, which is stored on the instance that way you don’t need to load it in themodel_fnand you can load it from a local path already.
2023-02-21T20:29:06Z
Hi@blackknight467I had this issue too last week and I worked around it by providing a custom inference script (which I needed anyway) and specifying the cache directory when downloading the model like so:model = AutoModelForSeq2SeqLM.from_pretrained(model_name, cache_dir="/tmp/model_cache/")Not ideal, but it did the trick.But I agree that this should work out of the box without any workaround. I might dig a bit deeper into this and let you know what I find.CheersHeiko
2023-02-21T23:07:06Z
That worked perfect!  Thanks!