Loading inference.py separately from model.tar.gz
Hi,I’m deploying some inference endpoints on Sagemaker using the HuggingFace Inference Toolkit, and am overriding some of the default methods (model_fn and predict_fn) as described here:huggingface.coDeploy models to Amazon SageMakerWe’re on a journey to advance and democratize artificial intelligence through open source and open science.This works, but the problem is that when I want to test changes to myinference.py, it is very slow and cumbersome to create and upload a newmodel.tar.gzfile that includes the newinference.pyscript.Is it possible to provide theinference.pyscript separately from the compressed model archive?
2023-03-22T10:47:26Z
No, there is sadly not way except that you can test your inference.py locally or load the model in the model_fn
2023-03-22T11:04:20Z
It is possible to give only the inference.py script if you have your model.tar.gz in s3. Not sure if it works with hub.When you create the HuggingFaceModel() object, give it source dir (local folder where inference.py script is), entry point (inference.py) and model_data (s3 url).Then next time you do HuggingFaceModel.deploy() it will use the inference script from your local folder and the model from s3.
2023-03-22T12:39:13Z
augustindal:It is possible to give only the inference.py script if you have your model.tar.gz in s3. Not sure if it works with hub.Thats true but behind the scenes a newmodel.tar.gzis built and used.
2023-06-05T08:47:51Z
liammc:it possible to provide theinference.pyscript separately from the compressed model archive?Where should the inference.py file be inside the model tar ball? Should it be inside the subdirectory “code”  or should it be immediately inside the tar ball as a file?