Support for exporting generate function to ONNX?
I was wondering if huggingface provided any support for exporting the generate function from thetransformerslibrary to ONNX?Mainly, I was trying to create an ONNX model using a GPT2 style transformer in order to speed up inference when generating replies to a conversation.I see there’s some support for exporting a single call to GPT2, but not the entire for loop used in greedy decoding/beam search/nucleus sampling etc.
2022-08-11T00:14:20Z
Take a look at huggingfaceoptimum, but not all models are supported.Documentation.
2022-08-11T18:43:40Z
Thanks for the response@guillermogabrielli. I’ve already taken a cursory look at optimum, however all the export scenarios seem pretty targeted. I could not find a way to export an LM + generalized decoder.
2022-12-07T18:11:33Z
Hi@nifarn, could you elaborate on what you would like to see that is not available in Optimum for encoder-decoder models?
2023-02-02T11:00:59Z
Hi@fxmartyI have the same demand.I want to export both the model and the generation strategy. The model architecture is similar to GPT with slight modifications. And the generation strategy, including greedy and beam search, requires pytorch to traceifandfor.I founda demo to export BART along with BeamSearch into ONNX. But it’s outdated.image1472×346 34.8 KBAny suggestion on how to achieve this in Optimum?@nifarnCould you share you final solution?Any help is appreciated!
2023-02-06T12:10:32Z
Hi@in-certo, this is a very reasonable feature request indeed. The issue is tracked atMake ORTModel PyTorch free · Issue #526 · huggingface/optimum · GitHubI very much doubt that transformersgenerate()will ever be jit scriptable. Therefore, the solution would probably be to rewritegenerate()and greedy and beam search for them to be jit.scriptable. The other option is to use ONNX RuntimeGreedySearchandBeamSearchops, but those are usable only with CPUExecutionProvider and CUDAExecutionProvider (useless if you want to use TensorRT for example).You could torch.jit.trace the model itself, and then torch.jit.script the generate method.
2023-02-08T07:14:36Z
Hi@fxmartyThanks for your suggestion.I have some questions on how to convert a generation model to TorchScript.Take GLM as an example. The signature of its forward function contains an Optional[Tensor] variablemem.  The code can be found here:modeling_glm.py · BAAI/glm-10b at main (huggingface.co).If we usetorch.jit.tracefunction to convert it to TorchScript. How can we trace two different forward passes? (One takes a realmemvariable which is a List[Tensor], and the other takes aNone)
2023-02-08T12:56:24Z
For generation, Ithinkyou will not be able to obtain a meaningful ScriptModule usingtorch.jit.trace. This is because the number of generation steps and early stopping depend on which tokens are generated, the number of loops is dynamic.torch.jit.tracewould just hardcode the number of loops from the sample example provided during tracing, but this is not useful for other samples.If we usetorch.jit.tracefunction to convert it to TorchScript. How can we trace two different forward passes?Normally you should just provide the model with a different set of inputs that trigger the controlflows you would like to use.