Export a BetterTransformer to ONNX
Hi,I wonder if it’s possible to export a BetterTransformer to ONNX in order to take advantage of the optimizations done by both. I gave it a shot and I encountered the below error:UnsupportedOperatorError: Exporting the operator 'aten::_nested_tensor_from_mask' to ONNX opset version 13 is not supported. Please feel free to request support or submit a pull request on PyTorch GitHub: https://github.com/pytorch/pytorch/issuesseems that there aren’t a lot of discussions on the internet and thus I wanted to post a question here. Thanks!
2023-05-15T12:13:13Z
Hi@dfangish, here is the list of ONNX-supported ATen operators:ONNX supported TorchScript operators — PyTorch 2.0 documentationSearching this page for_nested_tensor_from_maskshows that it is not supported yet (same for_transformer_encoder_layer_fwdwhich would also be needed).They provide a small guide if you would like to add support for these operators:torch.onnx — PyTorch 2.0 documentation
2023-08-09T11:40:11Z
Hi@dfangish, I am facing the same. Did you solve this issue (writing custom op, etc…) ?
2023-08-11T15:15:57Z
Hi@dfangish@EmreOzkose, this is unfortunately not possible. BetterTransformer relies on optimizations from PyTorch through custom CUDA kernels, which can not be exported to ONNX.