Torchscript Example for BERT
I am looking at the example for torchscripting BERT-like models here:Exporting 🤗 Transformers Models. I have a basic question about the dummy inputs being passed for tracing which don’t make obvious sense to me.The input passed is a list containingtoken_idsandsegment_ids(ortoken_type_ids) which torchscript will unpack. Now,BertModel.forward()expectsinput_idsandattention_maskas the first and second arguments respectively. So, how why issegment_idsbeing passed as the second argument for both tracing and later on for inference with the loaded torchscripted model? Does it somehow work because of the flagtorchscript=Truethat’s passed when instantiating the model? If so, how does it work?
2022-01-27T08:34:03Z
cc’ing@lewtunhere
2022-02-01T00:35:38Z
@lewtunany insights here?
2022-02-04T10:29:52Z
Hey@hikushalherethanks for raising this issue! This looks like an error in the guide and the only reason the code runs is because the tensor used forsegment_idsis similar to whatattention_maskshould be.Thetorchscript=Trueflag is used to ensure the model outputs are tuples instead ofModelOutput(which causes JIT errors).Would you like to open a PR to fix the guide?