Generate() method for models converted to torchscript
I’ve been experimenting exporting git-large-coco to torchscript and with a minor adjustment to the transformers library this seems to work. HoweverGitForCausalLMuses thegenerate()function, not just just a plain model() invocation, so I’m stuck on how to use the torchscript version of the model.Am I correct that I would need to reimplement whatever generate() is doing if I want to use the torchscript model? I’d eventually like to run it in a non-python environment, but for now I’m only experimenting.(edit: I did findGenerate without using the generate methodwhich give me some clues, but I’m not getting the correct output yet)Can I also load the torchscript model back into transformers just to see if the model behaves the same?
2023-08-01T06:58:12Z
Were you able to make any progress on this? I tried implementing the code from the post you have linked. But it is much slower than the generate( ) method.
2023-08-01T08:04:17Z
No not really. I did try to implement it but I think the model I’m using is not using greedy matching or something and the results weren’t similar. Still not sure why though.Then I found out I messed up the torchscript conversion and that it’s probably not going to work for what I’m trying to do, so I didn’t pursue it further.