Error deploying BERT on SageMaker
I fine-tuned BERT for text-classification on a custom dataset using HuggingFace and Tensorflow, and now I’m trying to deploy the model for inference through SageMaker. I followedthis HuggingFace tutorialbut I get the following error. I spent a while looking through theSageMaker HuggingFace documentationto no avail. The error says that model_uri is set to None, but model_uri is not a parameter that I can pass, and I just want it to pull my model from the HuggingFace Hub.I also tried downloading the model from the Hub, zipping it, uploading it to S3, and passing model_data=“model.tar.gz”, but that didn’t work either.image1885×856 134 KBAny help would be greatly appreciated!
2021-07-16T15:23:08Z
Resolved: I just needed to add an image_uri!
2021-07-21T07:06:13Z
Hey@wsunadawong, I moved your post into the Amazon SageMaker category in the forum. I didn’t catch it when you posted it.Quick question regarding your issue. Did you need to addimage_urito theHuggingFaceModelto run it? This should be the case with using the latest version ofsagemaker. Could you please share how you deployed it?
2021-07-21T12:33:09Z
Thanks for your help!At first, I addedimage_urito theHuggingFaceModelwhich worked. First, I tried usingimage_uris.retrieve(framework='huggingface',region='us-east-1', instance_type='ml.t2.medium',image_scope='inference',base_framework_version='tensorflow2.4.1'), but it gave an error that the image_uri could not be found, so then I went tothis list of imagesand chose the image uri with the following properties: TensorFlow 2.4.1 with HuggingFace transformers, inference, CPU, py37.Then I noticed that MultiDataModel requires launching from S3, so switched fromHuggingFaceModeltoModelin order to test pulling the model from the S3 bucket. Now that I look at it, it’s quite possible the model isnotusing the S3 model because I’m still passing inenv=hub. (Could that explain why theModelworks butMultiDataModeldoesn’t?)image1445×661 37.9 KB
2021-07-21T15:39:32Z
Update: usingModelgives the same error asMultiDataModelwhen I removeenv=hubas a parameter. I’m confused about this error, because my counterargument.tar.gz does contain a config.json file.image971×1149 75 KB
2021-07-22T09:18:21Z
I saw you usedfrom sagemaker.model import Model. We created a model classHuggingFaceModel, which you can use with out providing animage_uri.Documentation to this is here:Hugging Face — sagemaker 2.109.0 documentationandDeploy models to Amazon SageMakerand a small code snippet herefrom sagemaker.huggingface import HuggingFaceModel

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
    transformers_version='4.6',
    tensorflow_versuin='2.4',
    py_version='py37',
    model_data='s3://my-trained-model/artifcats/model.tar.gz',
    role=role,
)
# deploy model to SageMaker Inference
huggingface_model.deploy(initial_instance_count=1,instance_type="ml.m5.xlarge")We currently have an image fortensorflow 2.4.1.I got the model to work with the following snippet.image2206×1198 226 KBHow does yourmodel.tar.gzlook like?
2021-07-22T11:53:13Z
Yes, so I usedHuggingFaceModelbefore, and it did work successfully! The problem is that I want to run multiple models on the same endpoint. Unfortunately, I don’t know how to add aHuggingFaceModelto aMultiDataModel, which is why I was using a plain oldModelinstead. The only way I know how to add a model is by adding a path to the S3 bucket withadd_model(model_data_source, model_data_path). (Here, I cannot specify that I want the model to be aHuggingFaceModel, so I assume it defaults to a regularModel, which is why it fails?) Are you able to get aMultiDataModelrunning with multipleHuggingFaceModels?
2021-07-22T11:55:28Z
Mycounterargument.tar.gzis just a zipped version of my HuggingFace git repo:image858×616 36.3 KB
2021-07-22T12:56:17Z
Let’s move everything related toMultiDataModelto thisModel works but MultiDataModel doesn't - #10 by dan21cso we can discuss it there.If normal deployment works we can “close” this.Or does normal deployment with yourmodel.tar.gznot work?Could you share how you created the archive ofcounterargument.tar.gz. Sometimes the structure is ismodel.tar.gz
   - model
        - config.jsonThat way there is an extra folder and SageMaker might not recognize it.
2021-07-22T17:13:04Z
I don’t think normal deployment is working for me. I tried running the code snippet you sent, and it gave me an error about needing to define a TASK.image1092×935 63 KBWhen I added the HF_TASK in the env parameter, I got the same error as with the MultiDataModel: not being able to find the config.json.image1231×1042 69.1 KBI created the archive by cloning my HuggingFace repo, and runningtar -czvf counterargument.tar.gz counterargument_hugging. Here’s the structure:counterargument.tar.gz
    - config.json
    - tf_model.h5
    - tokenizer.json
    - special_tokens_map.json
    - tokenizer_config.json
    - vocab.txtI tried rerunning with themodelsubfolder structure that you suggested, but it just gave the same “config.json” error unfortunately.
2021-07-23T08:52:40Z
Hey@wsunadawong,the first image you shared is using yourmodel_dataand not thehubconfiguration.Could you try it using thehubconfiguration if this works then thecounterargument.tar.gzneeds to be wrong.Could you try creating thearchivewith the following steps?Download the modelgit lfs install
git clone https://huggingface.co/wsunadawong/counterargument_huggingCreate a tar filecd counterargument_hugging
tar zcvf model.tar.gz *Upload model.tar.gz to s3aws s3 cp model.tar.gz <s3://mymodel>
2021-07-23T15:58:53Z
That worked! The MultiDataModel works now! When I zipped the first time, I included the folder withtar zcvf model.tar.gz counterargument_huggingwhen I should’ve just included the foldercontents.Thank you so much for your help.
2021-11-19T06:19:58Z
Hi@wsunadawongand@philschmidI am facing an error when deploying a fine-tuned BERT model on sagemaker. I have trained the model already and the model is in S3 (model.tar.gz). When I try running the snippet of code below, I get an error calling thedeploy(...).  →TypeError: expected str, bytes or os.PathLike object, not NoneTypefrom sagemaker.huggingface.model import HuggingFaceModel
import sagemaker 

role = sagemaker.get_execution_role()

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
    model_data="s3://sagemaker/huggingface-pytorch-training/output/model.tar.gz",  # path to your trained sagemaker model
    role=role, # iam role with permissions to create an Endpoint
    transformers_version="4.6", # transformers version used
    pytorch_version="1.7", # pytorch version used
    py_version="py36", # python version of the DLC
)Then deploy# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
   initial_instance_count=1,
   instance_type="ml.m5.xlarge"
)And this is the error---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-36-da06cee412a2> in <module>
      2 predictor = huggingface_model.deploy(
      3    initial_instance_count=1,
----> 4    instance_type="ml.m5.xlarge"
      5 )

/opt/conda/lib/python3.6/site-packages/sagemaker/model.py in deploy(self, initial_instance_count, instance_type, serializer, deserializer, accelerator_type, endpoint_name, tags, kms_key, wait, data_capture_config, **kwargs)
    761             if self._base_name is not None:
    762                 self._base_name = "-".join((self._base_name, compiled_model_suffix))
--> 763 
    764         self._create_sagemaker_model(instance_type, accelerator_type, tags)
    765         production_variant = sagemaker.production_variant(

/opt/conda/lib/python3.6/site-packages/sagemaker/model.py in _create_sagemaker_model(self, instance_type, accelerator_type, tags)
    315         Args:
    316             output_path (str): where in S3 to store the output of the job
--> 317             role (str): what role to use when executing the job
    318             packaging_job_name (str): what to name the packaging job
    319             compilation_job_name (str): what compilation job to source the model from

/opt/conda/lib/python3.6/site-packages/sagemaker/huggingface/model.py in prepare_container_def(self, instance_type, accelerator_type)
    269 
    270         deploy_key_prefix = model_code_key_prefix(self.key_prefix, self.name, deploy_image)
--> 271         self._upload_code(deploy_key_prefix, repack=True)
    272         deploy_env = dict(self.env)
    273         deploy_env.update(self._framework_env_vars())

/opt/conda/lib/python3.6/site-packages/sagemaker/model.py in _upload_code(self, key_prefix, repack)
   1136             utils.repack_model(
   1137                 inference_script=self.entry_point,
-> 1138                 source_directory=self.source_dir,
   1139                 dependencies=self.dependencies,
   1140                 model_uri=self.model_data,

/opt/conda/lib/python3.6/site-packages/sagemaker/utils.py in repack_model(inference_script, source_directory, dependencies, model_uri, repacked_model_uri, sagemaker_session, kms_key)
    413 
    414         _create_or_update_code_dir(
--> 415             model_dir, inference_script, source_directory, dependencies, sagemaker_session, tmp
    416         )
    417 

/opt/conda/lib/python3.6/site-packages/sagemaker/utils.py in _create_or_update_code_dir(model_dir, inference_script, source_directory, dependencies, sagemaker_session, tmp)
    461             os.mkdir(code_dir)
    462         try:
--> 463             shutil.copy2(inference_script, code_dir)
    464         except FileNotFoundError:
    465             if os.path.exists(os.path.join(code_dir, inference_script)):

/opt/conda/lib/python3.6/shutil.py in copy2(src, dst, follow_symlinks)
    260     """
    261     if os.path.isdir(dst):
--> 262         dst = os.path.join(dst, os.path.basename(src))
    263     copyfile(src, dst, follow_symlinks=follow_symlinks)
    264     copystat(src, dst, follow_symlinks=follow_symlinks)

/opt/conda/lib/python3.6/posixpath.py in basename(p)
    144 def basename(p):
    145     """Returns the final component of a pathname"""
--> 146     p = os.fspath(p)
    147     sep = _get_sep(p)
    148     i = p.rfind(sep) + 1

TypeError: expected str, bytes or os.PathLike object, not NoneTypeI opened themodel.tar.gzlocally and this is the content of the folder:Screen Shot 2021-11-19 at 03.19.45812×178 22.2 KBAny help is much appreciated!
2021-11-19T07:57:34Z
Hey@cfloressuazo,Thank you for opening the thread and providing and the information.For looking at your folder structure ofmodel.tar.gzI see that you are missing atokenizers. How did you save your model + tokenizer in your training script?
2021-11-19T13:37:36Z
Hey@philschmid, thanks for the quick answer!I am using the train.py script that is part of the tutorial from Huggingface. I’m pasting it in here:from transformers import AutoModelForSequenceClassification, Trainer, TrainingArguments
from sklearn.metrics import accuracy_score, precision_recall_fscore_support
from datasets import load_from_disk
import random
import logging
import sys
import argparse
import os
import torch

if __name__ == "__main__":

    parser = argparse.ArgumentParser()

    # hyperparameters sent by the client are passed as command-line arguments to the script.
    parser.add_argument("--epochs", type=int, default=3)
    parser.add_argument("--train-batch-size", type=int, default=32)
    parser.add_argument("--eval-batch-size", type=int, default=64)
    parser.add_argument("--warmup_steps", type=int, default=500)
    parser.add_argument("--model_name", type=str)
    parser.add_argument("--learning_rate", type=str, default=5e-5)

    # Data, model, and output directories
    parser.add_argument("--output-data-dir", type=str, default=os.environ["SM_OUTPUT_DATA_DIR"])
    parser.add_argument("--model-dir", type=str, default=os.environ["SM_MODEL_DIR"])
    parser.add_argument("--n_gpus", type=str, default=os.environ["SM_NUM_GPUS"])
    parser.add_argument("--training_dir", type=str, default=os.environ["SM_CHANNEL_TRAIN"])
    parser.add_argument("--test_dir", type=str, default=os.environ["SM_CHANNEL_TEST"])

    args, _ = parser.parse_known_args()

    # Set up logging
    logger = logging.getLogger(__name__)

    logging.basicConfig(
        level=logging.getLevelName("INFO"),
        handlers=[logging.StreamHandler(sys.stdout)],
        format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
    )

    # load datasets
    train_dataset = load_from_disk(args.training_dir)
    test_dataset = load_from_disk(args.test_dir)

    logger.info(f" loaded train_dataset length is: {len(train_dataset)}")
    logger.info(f" loaded test_dataset length is: {len(test_dataset)}")

    # compute metrics function for binary classification
    def compute_metrics(pred):
        labels = pred.label_ids
        preds = pred.predictions.argmax(-1)
        precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average="binary")
        acc = accuracy_score(labels, preds)
        return {"accuracy": acc, "f1": f1, "precision": precision, "recall": recall}

    # download model from model hub
    model = AutoModelForSequenceClassification.from_pretrained(args.model_name)

    # define training args
    training_args = TrainingArguments(
        output_dir=args.model_dir,
        num_train_epochs=args.epochs,
        per_device_train_batch_size=args.train_batch_size,
        per_device_eval_batch_size=args.eval_batch_size,
        warmup_steps=args.warmup_steps,
        evaluation_strategy="epoch",
        logging_dir=f"{args.output_data_dir}/logs",
        learning_rate=float(args.learning_rate),
    )

    # create Trainer instance
    trainer = Trainer(
        model=model,
        args=training_args,
        compute_metrics=compute_metrics,
        train_dataset=train_dataset,
        eval_dataset=test_dataset,
    )

    # train model
    trainer.train()

    # evaluate model
    eval_result = trainer.evaluate(eval_dataset=test_dataset)

    # writes eval result to file which can be accessed later in s3 ouput
    with open(os.path.join(args.output_data_dir, "eval_results.txt"), "w") as writer:
        print(f"***** Eval results *****")
        for key, value in sorted(eval_result.items()):
            writer.write(f"{key} = {value}\\n")

    # Saves the model to s3; default is /opt/ml/model which SageMaker sends to S3
    trainer.save_model(args.model_dir)
2021-11-19T15:01:51Z
cfloressuazo:trainer = Trainer(
        model=model,
        args=training_args,
        compute_metrics=compute_metrics,
        train_dataset=train_dataset,
        eval_dataset=test_dataset,
    )Theretokenizeris missing and when you then calltrainer.save_modelit only saves the model and not the tokenizer as well.If you take a look at:https://github.com/huggingface/notebooks/blob/87cde801e765caf5936808c61d4edd62cc32abf1/sagemaker/01_getting_started_pytorch/scripts/train.py#L58The tokenizer is loaded and passed to the Trainer to save it as well.Can you share the link where you found this?
2021-11-19T15:29:50Z
I see! It makes total sense. I copied and pasted without any modifications the train.py script in this blog post from HuggingfaceThe Partnership: Amazon SageMaker and Hugging Face. Maybe that tutorial is outdated or requires some changes*Another question that I have is that I don’t defined anywhere the way the data is passed to the model for inference. I haven’t got to that point yet, but I am assuming the data should be passed as a json with a key asinputsand value as array of the text? Something like this:data = {'inputs': ['text_1', 'text_2', ...]}. And that is regardless of the dataset and its format that I have used?Thank you again!
2021-11-22T08:47:09Z
Thank you! i ll fix it there.For inference, theHugging Face Inference Toolkitis used, which is easily speaking a sagemaker compatible wrapper around thepipelinesobject with an API structure similar to the Inference API. You can find more documentation here:Reference