Variable length batch decoding
Hi All,Just want to know, is there any way to batch decode variable length sentences.For example [S1, S2] , where S1 has 5 words abd S2 has 10 words . Can we decode it using GPT2 , BERT etc?
2020-10-04T16:16:06Z
Hi@s4sarathWhat do you mean by decoding, as in decoding the generated tokens by GPT2  or making predictions on a batch of sequences ?
2020-10-04T16:31:34Z
By Decoding I mean, generate sequence of tokens.@valhalla.
2020-10-04T16:36:36Z
Alltokenizersoffer this functionality, just pass the list of seqs to ittokens = tokenizer([s1, s2])["input_ids"]by default it’ll pad all the seqs to the maximum length in the batch if they are of different length. You can find more detailed info in thisguide
2020-10-04T16:41:18Z
@valhallathanks for this. I  have seen that. But when I try the pad token in gpt2 it dint work as expected.
2020-10-04T16:47:20Z
There is no pad token for GPT, you can manually set theeostoken aspadtoken usingtokenizer.pad_token = tokenizer.eos_token
2020-10-05T03:03:41Z
Hi@valhalla.  Thanks for suggestion . I have done as per you say. This is the same thing I have tried before.  But as you see in screenshot, the results of variable batch sentences does not produce correct results.image3266×1424 404 KB
2020-10-05T09:23:25Z
As this is auto regressive model which predicts next token based on previous tokens, it might not generate correct tokens when there are eos in the text.I thought you were asking about batching at training time. Sorry about the misleading answer.Right nowgeneratedoes not support batched generation for gpt2.Pinging@lysandre
2020-10-05T11:53:33Z
No problem@valhalla. Appreciate your response.I have implemented this feature locally.Off topic, would like to know what is your thoughts on this.Why Tensorflow Models are way slower than Pytorch models, for autoregressive modeling?🤗TransformersHi, 
I was experimenting with many models include GPT2, T5 etc. But it seems like Tensorflow models are too slow for same type of generation comparing to Tensorflow, whether it is greedy, beam etc . 
Any specific reasons for this? 
Thanks
2020-10-05T12:28:12Z
Not a TF userGreat, can you share your fix if possible, lots of other people are interested in batched prediction for GPT.
2020-10-05T13:18:44Z
I have implemented in TF2.0 . I had to make quite a bunch of changes to make it work. Will share it.
2024-03-31T13:27:47Z
Hi, is there any updates? Thanks!