How to optimize ONNX seq2seq model?
Hi!I am trying to improve the performance ofsshleifer/distilbart-cnn-12-6summarization model using Optimum. Recently tested with other classification models and just by converting to ONNX I was getting an inference performance gain, but seems not to be the case with seq2seq models.I would like to know the process to optimize this kind of models, because the model is splitted into multiple parts and not supported by the regular optimizer.You can have a look at thisnotebook gistwhere I was trying to optimize each of the ONNX files produced after saving while measuring the performance vs the standard summarization pipeline.Thank you
2022-07-27T08:12:48Z
Hi@pablojs,To optimize a seq2seq model, you should first export it to the ONNX format usingORTModelForSeq2SeqLMand then apply optimization on each of its component (encoder,decoderanddecoder_with_past). We are currently working on the refactorization of theORTOptimizerin order to simplify its usage, you can follow the progress in#294. You might also be interested in applying dynamic quantization on your model with theORTQuantizer(refactorization in#270)
2022-08-25T08:24:17Z
Hi@Z3K3, let’s move this discussion toOptimize an ONNX Seq2Seq modelas you are describing the same problem there. Please don’t cross-post the same question in multiple topics in the future as it makes things difficult to track.