How to pass pipeline parameters when using sagemaker DataSerializer?
Hi folks, we’re trying to deploy an ASR model to sagemaker, but getting hung up on how to pass pipeline parameters to the endpoint when usingDataSerializer(as seems to be necessary).For example, to deploy and call an ASR model (in this case HUBERT), we can do it as:# create a serializer for the data
audio_serializer = DataSerializer(content_type='audio/x-audio') # using x-audio to support multiple audio formats

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
	initial_instance_count=1, # number of instances
	instance_type='ml.m5.xlarge', # ec2 instance type
    serializer=audio_serializer, # serializer for our audio data.
)

res = predictor.predict(data = "sample1.flac")

print(res)

{'text': "GOING ALONG SLUSHY COUNTRY ROADS AND SPEAKING TO DAMP AUDIENCES IN DRAUGHTY SCHOOLROOMS DAY AFTER DAY FOR A FORTNIGHT HE'LL HAVE TO PUT IN AN APPEARANCE AT SOME PLACE OF WORSHIP ON SUNDAY MORNING AND HE CAN COME TO US IMMEDIATELY AFTERWARDS"}So far so good, but in our case we’re dealing with longer audio files, and need to be able to pass thechunk_length_sandstride_length_sparameters.If you’re not using a serializer, it appears to be as simple as e.g.payload = {"inputs": input, "parameters": params}

response = sagemaker_runtime.invoke_endpoint(
    EndpointName=endpoint_name,
    ContentType='application/json',
    Body=json.dumps(payload)
)But because we’re using a serializer, we have to pass e.g.predictor.predict(data = "sample1.flac")and this does not seem to provide a way to include pipeline parameters like we need?For example, if we try:payload = {
    "inputs": "sample1.flac",
    "parameters": {
        "chunk_length_s" : 5
    }
}
res = predictor.predict(payload)We receive error messageValueError: Object of type <class 'dict'> is not Data serializable. This makes sense since DataSerializer expects a file path or raw bytes.So, how do we solve this? How do we correctly pass audio data into the sagemaker endpoint but also include pipeline parameters likechunk_length_sandstride_length_sso that the model operates correctly?Perhaps@philschmidor@marshmellow77would be willing to chime in?Help!
2022-11-29T22:21:15Z
Given that SageMaker Hugging Face Inference Toolkitbuilds on top of the pipeline feature, I took a look at thepipeline documentation for ASR, and it seems to me that parameters likechunk_length_sandstride_length_sare specified whencreatingthe pipeline, not at every inference request. I don’t have enough experience with ASR to say if that makes sense or not, but that’s what it looks like to me.Now, how to fix your problem with that information? Again, I have very little experience with ASR workloads, but at the very least I would think you could create a custom inference script, create and use the ASR pipeline in that script and pass the parameters to the endpoint when creating it with thedeploy()method via theenvdictionary. Should be sth along the lines ofmodel.deploy(..., env={'chunk_length_s': 5, 'stride_length_s': 10}, ...)and in the inference script:def model_fn():
    pipe = pipeline("automatic-speech-recognition", chunk_length_s=os.environ('chunk_length_s'), ...I haven’t tested this but maybe give it a try? Or, at the very least, I hope it sparks some other ideas how to go about thisCheersHeiko
2022-11-29T22:45:50Z
Thanks for the quick reply@marshmellow77! and that’s an interesting point about being able to set those parameters at pipeline creation and not just at inference time (to my knowledge you can set them at each request as outlined here:Making automatic speech recognition work on large files with Wav2Vec2 inTransformers). But I haven’t tried whether they can be set at pipeline creation as well, so I’ll be sure to test that.If weareable to set ASR pipeline parameters upon model deployment is there a way to do that without a custom inference script. i.e. Is there a way to pass pipeline-specific parameters to sagemaker at model deployment? Would love to avoid custom inference script if possible for speed of iteration but we’ll definitely go that route if it’s the only option.Something like (this is obviously wrong, just an example):# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
	initial_instance_count=1,
	instance_type='ml.m5.xlarge',
	pipeline_kwargs={'chunk_length_s': 20, 'stride_length_s': 5}, # can we do something like this?
	serializer=audio_serializer,
)e.g. Assuming the ASR pipeline can acceptchunk_length_sandstride_length_sat model instantiation, is there a way to pass those to either theHuggingFaceModelobject or in themodel.deploymethods themselves?As an aside: if these parameters actually cannot be set at pipeline creation, is there a way for us to pass them somehow inpredictor.predictor would we have to use a custom inference script at that point? Something like (again, this is wrong just an example):res = predictor.predict(data = "sample1.flac", pipe_kwargs={'chunk_length_s': 20, 'stride_length_s': 5})
2022-11-30T15:56:49Z
Yes, you’re right, you can also provide those parameters at inference request. I should have looked into thecodefirst before blurting out my ignoranceShows you how little I know about ASRBut it should also be possible to set them at pipeline creation according tothis.Looking at the code of the SM Python SDK and the SM Huggingface Inference Toolkit (SMHFIT), it seems to me that indeed this is a use case that is not covered when calling thepredict()method:SMHFIT expects the input to come in the format of a dictionary:sagemaker-huggingface-inference-toolkit/handler_service.py at main · aws/sagemaker-huggingface-inference-toolkit · GitHubWhich is fine for text pipelines - for those the default serializer is JSON which does provide the input as the SMHFIT expects:sagemaker-python-sdk/serializers.py at master · aws/sagemaker-python-sdk · GitHubBut when it comes to Data Serializer, this does not happen:sagemaker-python-sdk/serializers.py at master · aws/sagemaker-python-sdk · GitHubNote that SM Python SDK is called first, and only after that the SMHFIT is calledNot sure what the best way forward is here, because I believe this means that even a custom inference script would not help here, because the exception is raised by the SM Python SDK, which is called before the SMHFIT is called …
2022-11-30T21:37:41Z
Hmm. Okay, so given that SMHFIT is built on top of HF pipelines, then in principle it should be possible to pass some kwargs somewhere at pipeline instantiation right?It looks like the actual HF pipeline is instantiated insidetransformers_utils.pyhere.Do you happen to know how I would get some **kwargs down to that method? Presumably given that the ASR parameters appear to be able to be set at pipeline creation, if there’s a way for me to pass thestride_length_sandchunk_length_sparams down through to that method above then my model would be properly set up for subsequent inference calls…It looks likehandler_service.pycalls theget_pipelinemethodherebut doesn’t look like it sends through any hyperparameters…Can theloadmethod be overridden?The docs only mentionoverridingmodel_fn,transform_fn,input_fn,predict_fn, andoutput_fn…Thanks again@marshmellow77!
2022-12-01T12:41:53Z
model_fn()is the right place for overriding the loading of the model, see also thisexample.I think the best way to pass on parameters to model instantiation is via os environment variables, as I tried to outlinehere.One more asset that might be valuable when trying to set this up could be theSM SSH Helper library. It allows you to SSH into your endpoint and see what is going in there, run debugger, etc.
2022-12-08T18:49:37Z
Just a quick update here@marshmellow77. I got this working using your original suggestion of instantiating the pipeline inmodel_fnusing the chunk and stride args there. It’s not perfect, because it still doesn’t allow me to dynamically choose different chunk or stride lengths at inference time (even though the pipeline itself allows these args), but it’s a step in the right direction and working for now. I’ll report back if I figure out an elegant way to pass params at inference time, presumably by subclassingpredict_fn(though I haven’t tried it yet).Thanks again for your help. Really appreciate it.
2022-12-08T20:14:23Z
Thanks for the update. Based from what I have seen in the Python SDK code for the DataSerializer I wasn’t able to figure out a way to pass on parameters - it seems to me the code just doesn’t account for that.If I have time this week or next I’m thinking of raising an issue and/or a PR for the same.But if you do manage to figure out a way, please update this thread, I’d be very interested too
2022-12-08T20:17:33Z
Another thought - have you considered using the Whisper API instead of the HF API in the endpoint? It seems that the Whisper API can deal with audio files longer than 30 secs:GitHub - openai/whisper: Robust Speech Recognition via Large-Scale Weak SupervisionThat way you wouldn’t have to provide any parameters at all
2022-12-08T21:17:33Z
ha, not a bad idea! Are you aware of any documentation on how to deploy Whisper API directly (to aws) rather than via HF API?
2022-12-09T09:23:51Z
I mean, it’s your inference script, you can do whatever you want in it as long as the required libraries ar installed (via therequirements.txtfile).So, as long as you make sure Whisper library is installled you can just load the Whisper model and use that for inference. Sth likeimport whisper

def model_fn(model_dir):
    model = whisper.load_model("large")
    return model

def predict_fn(data, model):
    result = model.transcribe(data)
    return result["text"]should work.I will try this out later today myself (because I’m also starting on a project that involves Whisper) and get back to you.
2022-12-09T18:29:31Z
So, this worked like a charm for me:import whisper

def model_fn(model_dir):
    model = whisper.load_model("large-v2")
    return model

def predict_fn(audio_bytes, model):
    audio_file = "tmp.mp3"
    
    # need to put the byte stream into a tmp file as model.transcribe() expects a file name
    with open(audio_file, "wb") as binary_file:
        binary_file.write(audio_bytes['inputs'])
        
    result = model.transcribe(audio_file)

    return {"detected_language": result["language"], "transcription": result["text"]}This is my requirements.txt file:transformers==4.25.1
git+https://github.com/openai/whisper.git
2022-12-09T22:47:41Z
Beautiful. Thanks! I’ll give it a shot.
2023-06-02T11:44:18Z
how do you make the predictions then?I get: TypeError: Object of type bytes is not JSON serializable.Can you show me how you set de predictor and deploy it?
2023-06-09T13:34:33Z
I tried tour custom predict_fn() and I get:ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received server error (500) from primary with message "[Errno 30] Read-only file system:Do you know how I can solve it?