Big `generate()` refactor
Soon we will merge a big refactor of thegenerate()method:https://github.com/huggingface/transformers/pull/6949All transformer models that have a language model head rely on thegenerate()method,e.g.Bart, T5, Marian, ProphetNet for summarization, translation, …GPT2, XLNet for open-end text generationThegenerate()method has become hard to read and more and quite difficult to maintain. The goal of the refactor is thus as follows:Remove the_no_beam_search_generationand_beam_search_generationmethods and replace them with the four different generation methods corresponding to the four different types of generation:greedy_search(corresponds to num_beams = 1, do_sample=False),sample(corresponds to num_beams = 1, do_sample=True),beam_search(corresponds to num_beams > 1, do_sample = False), andbeam_sample(corresponds to num_beams > 1, do_sample = True).The following philosophy has been adapted here: the originalgenerate()method is kept at 100% backwards compatibility (ping me on github under@patrickvonplaten, if your specific use case has nevertheless been broken by this PR). In addition each of the four specific generation methods can be used directly. The specific methods are as “bare-bone” as possible meaning that they don’t contain any magic pre-processing of input tensors.E.g.generate()automatically runs theinput_idsthrough the encoder if model is encoder-decoder model vs. for each specific generate method they have to be added asencoder_outputsto themodel_kwargs;input_idsare automatically created in case they are empty or automatically expanded fornum_beamsingenerate()vs. this has to be done manually for each specific generate method. The reason behind this design is that it will give the user much more flexibility for specific use cases ofgenerate()and improves maintainability and readability. The user should not be limited in any way when directly using the specific generate methods. It should therefore pave the way to allow backprop through generate, make beam generation much easier, makes it easier to write “higher level” generate functions as is done for RAG… For more information on this design please read the docs, look into the examples ofgreedy_search,sample,beam_searchandbeam_sample.All of the generate parameters that can be used to tweak the logits distribution for better generation results,e.g.no_repeat_ngram_size,min_length, … are now defined as separate classes that are added to aLogitsProcessorList. This has the following advantages: a) better readability b) much easier to test these functions c) easier to add new logits distribution warpers. A huge thanks goes tohttps://github.com/turtlesoupywho has had the original idea of this design in this PR:https://github.com/huggingface/transformers/pull/5420Move allbeam_searchrelevant code into its own generation_beam_search.py file and speed up beam search. Beam search has gained more and more in importance thanks to many new and improved seq2seq models. This PR moves the very difficult to understand beam search code into its own file and makes sure that thebeam_searchgenerate function is easier to understand this way. Additionally, all Python List operations are now replaced by torch.tensor operations which led to a 5-10% speed up forbeam_searchgeneration. This change improves speed, readability and also maintainability. New beam search algorithms, such ashttps://arxiv.org/abs/2007.03909, should now be easier to add to the library.Tests have been refactored and new more aggressive tests have been added. The tests can also be very helpful to understand how each of the methods work exactly. Check outtest_generation_utils.py,test_generation_beam_searchandtest_generation_logits_process.More docstring especially on beam search and logits processors to make generate more accessible and understandable to the user.TODO:Do the same refactor for TFCheck possibility of carrying gradients through generateAddGenerationOutputssimilar toModelOutputsthat allows to return attention outputs and hidden states
2020-11-03T03:07:02Z
Hi Patrick@patrickvonplaten,Thanks for your great work!! I have a question.I now almost finish translatingTFDPR, and currently also working on TFRag (I already madeTFRagModelworks like Pytorch, and I am investigatingTFRagTokenForGenerationright now) .Since the current code ofRagTokenForGenerationinvolves with both_no_beam_search_generationand_beam_search_generation, regarding the Refactor, I think I should just go ahead with the current version, and fix them later for refactor.Or would you suggest that I wait for Pytorch Refactor, or wait for both Pytorch & TF Refactor ?
2020-11-03T09:35:52Z
That’s amazing to hear that you are working on the TFRagModel ! do you already have a PR on this?I’d definitely say that you should go ahead with_no_beam_search_generationand_beam_search_generationin TF (it’ll take probably at least ~2,3 weeks until I start the TF refactor and > a month until I merge it). Feel free to tag me on the TFRagModel
2020-11-03T10:58:44Z
Hi Patrick, thank you for your suggestion.as Rag uses DPR’s pretrained weights as a submodule, so I made a TFDPR PR first. Now it’s around 25% progress on TFRag … I hope I can make a TFRag PR before EMNLP , and I will seek your suggestions on TFRag for sureJust in case you are interested in, my current draft with TFRagModel ishere
2020-12-22T14:32:01Z
Hello – on a hopefully related note, I saw in a recent Huggingface email that you had implemented Diverse Beam Search.  I was wondering, how is this applied and how can I use it?I have been using beam search with sampling in a summarization pipeline (via Bart-Large-cnn), and I was wondering how I can add more diversity to each beam generated? Thanks!
2020-12-23T02:34:49Z
Hi you can use the new diverse beam generation easily by just adding the two new parameters intogenerate().Two new parameters on diverse beam arenum_beam_groupanddiversity_penaltywherenum_beam_group <= num_beamsExampleoutputs = model.generate(input_ids, min_length=1, max_length=200,
                                  no_repeat_ngram_size=3, num_beams=5, early_stopping=True,
                                  do_sample=False, num_return_sequences=5,
                                  num_beam_groups=5, diversity_penalty=2.0
                                 )
2020-12-25T22:25:17Z
It only works withdo_sample=Falsehowever
2021-11-26T06:05:34Z
I just knew about diverse beam search, I have to try it for my abstractive text summarizer but unfortunately it works mostly extractive. I hope this diverse beam search could help to create abstractive summary little bit