Transformers / T5 , jit trace, script, quantize
Is it possible, when using TorchServe for inference, to improve the speed of inferencing T5 in specific (or transformers in general) by doing either:jit.tracejit.scriptquantizeAnd if possible, how?When I simply try to save/export a pretrained model using:traced_model = torch.jit.trace(model, (dummy_input_ids, dummy_attention_mask, dummy_decoder_input_ids))
	torch.jit.save(traced_model, "t5_small_traced.pt")I get an error message.
2023-04-18T18:55:40Z
Hey@ndvb! Did you solve it?
2023-04-18T20:02:32Z
@rahulbhalley, Nope