Variable num_predict in target_mapping for XLNet
I have a pretraining task for XLNet. One of the inputs for XLNetLMHeadModel istarget_mappingthat is of the shape(batch_size,num_predict,seq_len).I want to predict for all tokens in an input sentence, which meansnum_predictwill vary within a batch, for sentences of different length. This leads to error while building a data_loader in PyTorch. Can anyone suggest a workaround for this problem?Thanks
2021-01-02T08:18:06Z
I might be wrong here but I would assume that in the language modeling task, num_predict is actually the size of the vocabulary because for each mask you try to predict the highest probability token in the vocab (in MLM). Seq_len is the max length of sequences you want to be able to model. If a sentence is smaller then you just pad it.
2021-01-02T09:34:12Z
Dear@BramVanroy, thank you for the reply, butnum_predictis not the size of vocabulary. It’s the number of predictions to be made for that particular input sentence.seq_lenis not the issue because as you pointed out, it can be padded. But I am not sure if the same is true fornum_predict, hence this question
2021-01-02T09:58:00Z
You are absolutely right.But if you want to predict all tokens, can’t you just leavetarget_mappingto the default (None)? From the docs:If target_mapping is None, then num_predict corresponds to sequence_length.Or is your point that this mapping does not take into account the different sequence lengths in the batch? If that is the question, I cannot help with that. I do not have enough experience with XLNet. Perhaps someone else can chime in.