Predict function ignore parameters
Hey,I’m trying to deploy a Hugging Face Model (GPT-Neo) on the SageMaker endpoint. I followed the officialexampleand thisforum. but it seems that the generate function is totally ignoring my parameters (it generates just one word despite setting the min length to 10000!) Any idea what is wrong?My code:from sagemaker.huggingface import HuggingFaceModel
import sagemaker

role = sagemaker.get_execution_role()
# Hub Model configuration. https://huggingface.co/models
hub = {
	'HF_MODEL_ID':'EleutherAI/gpt-neo-1.3B',
	'HF_TASK':'text-generation'
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
	transformers_version='4.6.1',
	pytorch_version='1.7.1',
	py_version='py36',
	env=hub,
	role=role, 
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
	initial_instance_count=1, # number of instances
	instance_type='ml.g4dn.xlarge' # ec2 instance type
)

prompt = "Some prompt"

gen_tex = predictor.predict({
	"inputs": prompt,
    "parameters" : {"min_length":10,}
})

print(gen_tex[0]['generated_text'])
2022-01-28T10:45:26Z
Could you please update to the latest version? which would betransformers_version="4.12"andpytorch_version="1.9"and test it again?What is the output ofgen_tex[0]['generated_text']? Is the,in the “parameters” on purpose?
2022-01-28T11:20:27Z
Updated to the latest version but it still ignore the parameter:predictor.predict({
    'inputs': "Can you please let us know more details about your",
    'parameters': {"min_length":1000}
})output:[{'generated_text': 'Can you please let us know more details about your account?\n\nHello,\nI am interested in the above-mentioned company and I have read some very interesting articles about it. I am interested in starting the work. Please let me know if'}]```
2022-01-28T11:24:13Z
I guess it’s only for min_length since it works fine for max_length=3output:[{'generated_text': 'Can you please let us know more details about your research'}]
2022-01-28T12:24:22Z
And when you runmodel.generatewith your parameter in a colab or other environment with the sameTransformersversion it works?
2022-01-28T12:37:29Z
Yes, it works fine when I use it locally withfrom_pretrainedmodel.
2022-01-28T12:57:48Z
Hi Ali, what happens if you set themin_lengthand themax_lengthparameters explicitly? I’m asking because I believe the text generator uses themax_lengthparameter from the model configuration if you don’t set it explicitly. And if themax_lengthparameter in the config file is smaller than youmin_lengththe output would be truncated. So, just wondering if setting both (e.g.min_length=1000andmax_length=2000) helps?
2022-01-28T13:29:51Z
Thank you both for your help. That solved the problem@marshmellow77.
2022-01-28T13:46:42Z
Awesome, glad it worked. It’d be great if you could mark this thread as Answered/Solved - it would make it easier and quicker in the future for other users with the same problem to find the solutionCheersHeiko