ClientErro:400 when using batch transformer for inference
Hi everyone,I try to do sentiment analysis on a bunch of data and follow the example notebooknotebooks/sagemaker-notebook.ipynb at main · huggingface/notebooks · GitHuband below is my code:from sagemaker.huggingface.model import HuggingFaceModel


hub = {
 'HF_MODEL_ID':'cardiffnlp/twitter-roberta-base-sentiment',
 'HF_TASK':'text-classification'
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
env=hub, 
role=role, 
transformers_version='4.6',
pytorch_version="1.7", 
py_version='py36', 
)

# create Transformer to run our batch job
batch_job = huggingface_model.transformer(
 instance_count=1,
 instance_type='ml.p3.2xlarge',
 output_path=output_s3_path,
 strategy='SingleRecord')

# starts batch transform job and uses s3 data as input
batch_job.transform(
 data=input_s3_path,
 content_type="application/json",    
 split_type="Line")My input jsonl file is formatted asimage1485×231 19.8 KBThe model successfully infer when I feed with  first 10 rows  (totally this data is about 6k rows)but it throw errors when I expand to 100 rows.I’m super new to sagemaker and hugging face ,  can anyone tell me what I’m missing ? Thank you
2022-01-07T22:11:32Z
sorry it’s still me, just attach more pics so the context might be better understood.I heard BERT has limitation on text length so I truncated each line to 460 wordsimage1501×577 14.6 KB
2022-01-10T11:14:49Z
Hi miOmiO - did the truncation fix your problem? If not, would you mind sharing what error message you are getting?
2022-01-10T17:10:01Z
Hi@marshmellow77,   yes, truncation make some long text acceptable to the model, but not all of them,even I specify the length to 460 words.  I tried from 500 to 460.  I have no idea if I should  keep reducing this or other modification would help.image1590×458 38.5 KB
2022-01-10T17:27:38Z
@miOmiOyou have created a second thread:ClientError:400 when using batch transformer on sagemaker for inferencewhere i responded. Can this be closed? or those two different threads.
2022-01-10T19:47:03Z
Hi@philschmid, sure, please kindly close this post, I will update on that one. Thank you.
2022-01-11T07:11:09Z
Thanks for letting me know. Then here again the responseHey@miOmiO,Happy to help you here! to narrow down your issue. I think the first step would be to check if the dataset was created in the sample (notebooks/sagemaker-notebook.ipynb at master · huggingface/notebooks · GitHub) works or if it also errors out.Additionally, could you bump the version of theHuggingFaceModelto the latest one? Fortransformers_versionthat’s4.12.3and forpytorch_versionits1.9.1maybe this already solves your issue. You can find the list of available containers here:ReferenceAlso worth testing is to replace your model with a different model, e.g.distilbert-base-uncased-finetuned-sst-2-english
2022-01-11T19:03:19Z
Hi@philschmid,Thank you for the reply!  I check all three approaches :1 The inputs format comply with the json format from sample notebook.2 I use the latest version of of framework as suggested3 Also switch to different model to see if it’s specific model issue.It still prompts client error:400 , but I notice that as long as I truncated my each line of text to 60 words(randomly pick the number) , both two model work fine. Is this related to the text length limitation? If so, is there any way I can specify the length of text when building the  batch job?P.S. my data set is around 6k rows , some  rows have more than 512 words/tokens.
2022-01-11T20:31:51Z
Hi@miOmiO- text length limitation could be an issue here. Note that the length refers to the number of tokens, not the number of words. Because BERT models generally use subword tokenization it can happen that one word is split into 2 or more tokens. That is why even reducing the number of words to 460 sometimes might throw an error.To test this you could try to use the model row by row and see if the number of examples that fail correspond to the same ones in your batch job. If it is indeed the number of tokens that cause the model to fail you should be seeing an error message like"... sequence length is longer than the specified maximum sequence length for this model ..."If this indeed the source of error then it might be easiest to truncate the input sequence of tokens after the tokenization  (rather than the number of words before tokenization).Hope that helps.
2022-01-12T07:15:22Z
The modelcardiffnlp/twitter-roberta-base-sentimentdon’t has amax lendefined. I tried to reach out the authors but they haven’t responded. SeeAdd `tokenizer_max_length` to `cardiffnlp/twitter-roberta-base-sentiment` · Issue #13459 · huggingface/transformers · GitHubYou could “fork” the model → creating a new model repository and push the weights + add thetokenizer_configthentruncation:Trueshould work properly.
2022-01-13T04:47:22Z
Hi@marshmellow77, thank you for leading me into thought of sub-word tokenization .
2022-01-13T04:52:06Z
Hi@philschmid,  I got solution from another related post under your help as well :How are the inputs tokenized when model deployment? - Amazon SageMaker - Hugging Face ForumsAfter I switch to another model and remodify the input json file format like this:{“inputs”: “long sentence 2”, “parameters”: {“trucation”: true}},  the new model works well for me ( as long as it has ‘max_length’ attribute in tokenizer config file.)