Install custom python libraries in HuggingFace DLC
Hi,I have the source code for a custom data cleaning library; the code is private to my company. I usually install it from local source using something ‘pip install ./cleanse’. I know the requirements.txt file can be used to specify package versions for public libraries like pandas and NumPy but how can we install custom packages in the HuggingFace DLC?
2022-08-08T06:21:15Z
You can put./cleanseinto yousource_dirand then either addos.system("pip install ./cleanse")at the top of your training script or add the local path./cleanseinto arequirements.txtin thesource_dir.
2022-08-08T13:40:29Z
Hi, I’m actually using this library to clean in inference. Is there a place I could leverage the inference.py file inside the code/ directory in the model.tar.gz file? Or otherwise can I somehow integrate the into the preprocess script (using SKLearnProcessor)
2022-08-08T14:03:47Z
Sorry, i am not sure what you mean withIs there a place I could leverage the inference.py file inside the code/ directory in the model.tar.gz file?
2022-08-08T14:10:05Z
I’m using an inference.py file right now as per instructionsat the bottom of this pageto override defaults during model inference. I’m wondering how/if I can use the custom library within this file
2022-08-08T14:17:46Z
philschmid:You can put./cleanseinto yousource_dirand then either addos.system("pip install ./cleanse")at the top of your training script or add the local path./cleanseinto arequirements.txtin thesource_dir.^like that
2022-08-08T14:24:09Z
What is source_dir though? Like where is that argument specified
2022-08-08T14:35:52Z
How are you deploying your model currently? could you share the code snippet?source_dirrefers to theHuggingFaceModelof thesagemaker-sdk.
2022-08-08T14:43:13Z
# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
   model_data= model_uri,  # path to your trained sagemaker model
   role=role, # iam role with permissions to create an Endpoint
   transformers_version="4.17", # transformers version used
   pytorch_version="1.10", # pytorch version used
   py_version="py38", 
   env={ 'HF_TASK':'text-classification' }# python version of the DLC
)
2022-08-08T14:55:27Z
does themodel_uricontain acode/inference.py?  If so you need to put the requirements.txt /./cleanseinto thecode/directory.