SageMaker Inference for Model Tuned Elsewhere
Reading through the documentation for HuggingFace & SageMaker as we are evaluating it and found the following:Q: Which models can I deploy for Inference?A: You can deployanyTransformers model trained in Amazon SageMaker, or other compatible platforms and that can accomodate the SageMaker Hosting designany of the 10 000+ publicly available Transformer models from the Hugging FaceModel Hub, oryour private models hosted in your Hugging Face premium account!Is it possible to fine-tune a model elsewhere, outside of SageMaker Training, (for instance, just through a regular PyTorch training loop on a pretrainedtransformersmodel), and then deploy it for Inference without hosting it on an account?Would appreciate any pointers y’all can give on this.
2021-08-30T17:42:37Z
Hello@charlesatftl,Yes, you can fine-tunetransformersanywhere you want and use it in SageMaker for Inference.There are currently two options to use your model then:Push your model toModels - Hugging Faceand deploy it directory from there, see:Deploy models to Amazon SageMakerCreate amodel.tar.gzupload it to s3 and deploy it from there, see:Deploy models to Amazon SageMakerIn Addition to 2. here is the documentation on how to create amodel.tar.gzDeploy models to Amazon SageMaker
2021-08-30T18:16:12Z
Thank you! This is very helpful.
2021-09-02T15:06:06Z
Now running into issues when I attempt to use a model as you suggested with a modifiedmodel.tar.gz. Specifically, when I add acodedirectory withinference.pyandrequirements.txt, it starts throwing errors even if those are both empty files, saying that it cannot find the config. The same model with thecodedirectory removed works properly. Error is along the lines of “Can’t load config for … Make sure that: … is a correct model identifier listed onModels - Hugging Face… or … is the correct path to a directory containing a config.json file.”
2021-09-02T15:32:19Z
Update: I believe this was an error on my end. You’ve gotta be careful totarthe model with the right flags or else the archive won’t have the right structure (i.e. it’ll have the files nested in some relative path).