How to convert mT5 and ByT5 to ONNX format?
Hi,ONNX allows to compress transformers models and speed up the inference time on CPU and GPU.Who could share code / notebook to convert mT5 and ByT5 models to ONNX format?There is the libraryfastT5of@kirafor T5 conversion (great!) but it has not been updated to the latest version of transformers and therefore, it does not accept mT5 and ByT5 models.Thanks.List of topicos about this subject:Speeding up T5 inferenceBoost inference speed of T5 models up to 5X & reduce the model size by 3XQuestions on distilling [from] T5List of online documents about this subject:Exporting transformers modelsNotebook:onnx_t5.ipynbfrom@valhallaNotebook:onnx-export.ipynb
2021-11-09T15:23:00Z
Hi@pierreguillouI recently tried to use fastT5 with on a mT5 model and it worked great (regardless the warning), however you’ll need to havetransformers=="4.6.1".Are you getting any error ? Or are the outputs nonsense?Note: I changedT5ForConditionalGenerationtoMT5ForConditionalGenerationinonnx_exporter.pybut I still think it was working without this.
2021-11-09T16:12:08Z
Thanks@JulesBelveze. I guess you did the right choice with the change fromT5ForConditionalGenerationtoMT5ForConditionalGenerationinonnx_exporter.pyof fastT5 (but it means that we need to manually change the official fastT5 library…).In the case ofByT5, it is different as theByT5Tokenizeris not part oftransformers=="4.6.1".How to convert a ByT5 model to the ONNX format?
2021-11-09T16:16:05Z
Argh yes for theByT5it sounds like you’ll need to have the fastT5 library updated, which unfortunately doesn’t seem trivial
2021-12-22T19:36:12Z
fastT5 now works with transformers>=4.12.5 (this is the latest version I have tested). It also has some tweaks to the quantization settings that should speed it up a bit more.I can confirm it works with byt5 now, and it should work with mt5 as well (except for needing to change the ConditionalGeneration class as noted above). However,  byt5 is substantially slower due to both the larger encoder and needing one decoding step per characterEDIT: I just opened a PR to explicitly support mt5 as well. I did some limited testing and it seems to work.