How to Improve inference time of facebook/mbart many to many model?
If we tried to run translation service on facebook mbart many to many on cpu it take 9 secs to translate, how do we reduce the inference time further…
2021-03-02T08:04:31Z
Hi@Vimal0703, one idea could be to try quantizing the model’s weights to a lower precision datatype. See e.g. step 2 in this guide:Dynamic Quantization — PyTorch Tutorials 1.7.1 documentationThis usually gives you a 2-3x reduction in latency and model size
2021-03-15T12:02:36Z
Thank you it worked out, now we are able to quantize the model where the time takes around 1-2 seconds, but is there any way to decrease the time further…
2021-03-15T15:47:55Z
Hi@Vimal0703One more option to improve the speed is to useonnx_runtime, but at this moment we don’t have any tool/script which will let you import exportMBarttoonnx.I’ve written a script for exporting T5 toonnx, something similar can be used forMBartas wellSpeeding up T5 inference🤗Transformersseq2seq decoding is inherently slow and using onnx is one obvious  solution to speed it up. Theonnxt5package already provides one way to use onnx for t5. 
But if we export the complete T5 model to onnx, then we can’t use the past_key_values for decoding since for the first decoding step past_key_values will be None  and onnx doesn’t accept None input. Without past_key_values onnx won’t give any speed-up over torch for beam search. 
One other solution is to export the encoder and lm_head to on…Another option is, we have also ported theM2M100model, which is SOTA many-to-many translation model. Them2m100_418Msmaller thanMBart50, can give more speed-up.  Here’s an example of how to useM2M100,M2M100
2021-06-10T16:59:34Z
Are you able to share this script or settings for quantizing this model? I would greatly apreciate it as I’m also going through a similar undertaking!@valhallaNot to be greedy but was any one able to provide a script for this? Otherise I would like to work on this.
2022-10-04T13:31:43Z
@Vimal0703@addressoiccan you please share the script, I have similar task…