BART question, it seems that pretraining is not work for a small model?
What is your question?My task is to generate keywords from sentences.I pretrain a text-generation model. I mask the sentences’ tokens and predict the whole sentences’ tokens.Pretraining batch_size = 8 and step = 1000000I haven’t observed improvement from pretraining. BLEU score is 10.5 for not pretraining, BLEU score is 9.5 for pretraining.CodeI take the python code fromgithub.comgoogle-research/pegasus/blob/master/pegasus/models/transformer.py#L38from pegasus.layers import attentionfrom pegasus.layers import decodingfrom pegasus.layers import embeddingfrom pegasus.layers import timingfrom pegasus.layers import transformer_blockfrom pegasus.models import baseimport tensorflow as tffrom tensorflow.contrib import layers as contrib_layersclass TransformerEncoderDecoderModel(base.BaseModel):"""Transformer encoder+decoder.Notations:B: batch_size, I: max_input_len, T: max_target/decode_len, D: hidden_sizeV: vocab_size"""def __init__(self, vocab_size, hidden_size, filter_size, num_heads,num_encoder_layers, num_decoder_layers, label_smoothing,dropout):hidden_size = 512num_encoder_layers = 3num_decoder_layers = 3DiscussionThe task is to generate keyword from sentences.The keyword may not appear in the sentences.So input masked sentences to predict whole sentences, it is not benefit the keywords generation task.Input masked sentences to predict whole sentences, it do not have relation to the keywords generation task.Am I right? Is it the reason that pretraining do not improve the BLEU score?Thank you very much.
2020-07-29T08:49:07Z
With all due respect, you are asking a question on a forum dedicated to a specific librarytransformersby HuggingFace, but the question does not involve that library. In fact, you are using a completely different library. I am not sure if this is the right place for such questions.@sgugger
2020-07-29T09:06:50Z
I have changed the tag.
2020-07-29T13:25:33Z
On the research part of the forum, we welcome any general questions, though of course we would prefer you to use our models@sshleifermight have some answer as he is the Bart person on the team.
2020-07-29T14:21:42Z
guotong1988:So input masked sentences to predict whole sentences, it is not benefit the keywords generation task.Input masked sentences to predict whole sentences, it do not have relation to the keywords generation task.Am I right? Is it the reason that pretraining do not improve the BLEU score?Thank you very much.Definitely possible, there could also be a bug in your code. I  don’t have enough familiarity with your task to know what results to expect.
2020-07-30T01:41:44Z
Thank you. I am also using your models.
2020-08-03T03:10:01Z
1, I pad some zeros in the input tokens for multi sentences. The output positions of output tokens should be exactly same to the input tokens, which means I should keep the padding zeros in the output tokens.2, The pretraining time should be longer.