BART_LM: Odd Beam Search Output
Hi folks,Problem: fine-tuned model adopts peculiar behaviour with beam search.Specifically, beam search outputs include 2 bos tokens and exclude the first word token. I have double checked my data feed and the inputs look fine, with one bos and the first word token is present. Even more strange, there are no such problems with nucleus sampling, which suggests that this is particular to beam search.for example, here are two outputs from the same fine-tuned model;beam search output: <s><s>rek likes to play football. </s>
nuleus sampling output: <s> Derek likes to go to the football pitch. </s>Any clues?
2020-08-06T18:21:55Z
Hi,how did you preparelabelsanddecoder_input_ids?
2020-08-06T18:25:03Z
As the model is generating bos token my guess is you didn’t shift labels. For BART we need to preparelabelsanddeocder_input_idslike thistarget_ids = tokenizer("...")
decoder_input_ids = target_ids[:, :-1].contiguous()
labels = target_ids[:, 1:].clone()
2020-08-06T18:56:51Z
Hi@valhalla, double checked but there might be a hole - here is my training_step function (I use lightning) :def training_step(self, batch, batch_id):
        """see lighting docs. Need to make sure that first token is not ignored"""

        decoder_inputs = batch["target_ids"][:, :-1].contiguous()
        decoder_labels = batch["target_ids"][:, 1:].clone()
        decoder_labels[batch["target_ids"][:, 1:] == self.tokenizer.pad_token_id] = -100

        loss = self(source_ids=batch["source_ids"],
                    padding_mask=batch["padding_mask"],
                    decoder_inputs=decoder_inputs,
                    decoder_labels=decoder_labels)[0]

        return {"loss": loss}
2020-08-06T19:01:43Z
Pinging@sshleifer
2020-08-06T19:06:55Z
here’s my datasetgetitemfor some extra info:def __getitem__(self, index) -> dict:
        """Generates ONE sample of data, DataLoader takes care of batches. We return everything here to sacrifice speed
        for ease of coding / reading. Can make faster in time but not needed as of yet.
        :param index: ID of single file we want to load data from
        :return item: dictionary containing all possible permutations of data for all possible objectives, such that X
        is the dialogue, y is the summary, and s is the sep_token " TL;DR: ".
        """

        SAMsumID = self.ids[index - 1]

        # load source sample string
        # space added to beginning for tokenization purposes ie tok("name") !+ tok(" name")
        source_file_path = f"{self.root}/source/source_{SAMsumID}.txt"
        with open(source_file_path, "r") as source_file:
            dialogue = source_file.readline()
            if "t5" in self.arch: dialogue = "summarize: " + dialogue

        # load target sample string
        # space added to beginning for tokenization purposes ie tok("name") !+ tok(" name")
        target_file_path = f"{self.root}/target/target_{SAMsumID}.txt"
        with open(target_file_path, "r") as target_file:
            summary = target_file.readline()

        # we will return a dictionary for each getitem() call that is collated by dataloader
        item = {}
        input_tokens = self.tokenizer(dialogue, padding="max_length", truncation=True, add_special_tokens=True)
        target_tokens = self.tokenizer(summary, padding="max_length", truncation=True, add_special_tokens=True)

        # return necessary batch tensors
        item["SAMsumID"] = SAMsumID
        item["source_ids"] = torch.tensor(input_tokens["input_ids"])
        item["padding_mask"] = torch.tensor(input_tokens["attention_mask"])
        item["target_ids"] = torch.tensor(target_tokens["input_ids"])

        return item
2020-08-06T19:39:48Z
can you show an example where you callgeneratetwice with the same input_ids and model, but different kwargs, and get wierd beam search output?I suspect the issue is in generate kwargs/decoder_start_token_idif same model -> different outputs.
2020-08-07T08:25:55Z
Both models have the same, defaultdecoder_start_token_id, and I when I manually override this to any arbitrary token<a>, the beam output looks something like<a><s>rek likes to play football. </s>.EDIT: The below code has decoderskip_special_tokens=True, but this flag is false for above examples.def test_step(self, batch, batch_id):
        """Generate a summary and write it to a file while calculating rouge"""

        # We store the file id to help with consistent results across several GPUs
        SAMsumIDs = batch["SAMsumID"]

        # generate y_preds given seed_ids, placing kwargs in dict is handy for debugging
        if self.sampling == "beam":
            generate_kwargs = {"input_ids": batch["source_ids"],
                               "early_stopping": self.generation_early_stopping,
                               "no_repeat_ngram_size": self.no_repeat_ngrams,
                               "max_length": self.tokenizer.max_len,
                               "top_k": 50,
                               "num_beams": self.beam_size}

        elif self.sampling == "nucleus":
            generate_kwargs = {"input_ids": batch["source_ids"],
                               "max_length": self.tokenizer.max_len,
                               "top_k": 50,
                               "do_sample": True,
                               "top_p": 0.95}

        y_preds = self.model.generate(**generate_kwargs)
        # save generated hypotheses to hypo dest with the same name as the original file, but with a 0 prepended
        for file_id, hypo_token_ids in zip(SAMsumIDs, y_preds):
            prefix = "source_0" if self.anti else "hypo_"
            filename = f"{self.hypo_dest}/{prefix}{file_id}.txt"
            with open(filename, "w") as handle:
                hypo = self.tokenizer.decode(hypo_token_ids, skip_special_tokens=True)
                handle.write(f"{hypo}\n")

        return {"hypo_dialogues": None}
2020-08-07T14:32:12Z
Another update: everything runs smoothly for T5 with all other code unchanged
2020-08-07T14:44:48Z
Maybe this line is causing an issue?https://github.com/huggingface/transformers/blob/3be2d048848df43e2dacb9127d2278443157ba78/src/transformers/modeling_bart.py#L1076
2020-08-07T14:57:06Z
@sshleifer, why do we need to manually set thebos_token_idhere, I think thegeneratefunction handles that here alreadyhttps://github.com/huggingface/transformers/blob/master/src/transformers/generation_utils.py#L402
2020-08-07T19:23:32Z
It’s copied from fairseq, but the basic idea is that you want to know the real probability of BOS, not just force it with probability 1. If I recall (open to being disproved), removing it breaksRUN_SLOW=1 pytest tests/test_modeling_bart.py
2020-08-08T18:32:30Z
I look forward to taking a closer look at this breakage… eventually. Thank you both for the help!
2020-08-10T12:26:27Z
@chrisdoyleIE,so the reason for twobostokens in beam search is that,herethe generate function sets thedecoder_start_token_id(if not defined thenbos_token_id, which<s>for BART) as the prefix token,andthisforces the generation ofbos_token(<s>) when the current length is one.So I also faced the same issue for my bart model fine-tuned for question generation when using beam search. This is what it produced for one of the inputs.['<s><s> created Python in 1991?']So when I just returned logits as it is fromadjust_logits_during_generationinstead of forcing anotherbos_tokenit generated this['<s>Who created Python?']Which is what I expected.But as@sshleifersaid, when I modifiedadjust_logits_during_generationas said above it breaks theRUN_SLOW=1 pytest tests/test_modeling_bart.pybecause the generated summaries don’t match the fairseq ones.So when I looked at the configs of thebart-large-cnnandbart-large-xsum, it usedeos_token(</s>) as thedecoder_start_token_idand test only pass when the first two token are</s><s>(</s>is set as first token by.generateand<s>is set byadjust_logits_during_generation.So I suspected that the weird beam search output for my model is because of using<s>asdecoder_start_token_id, when I tried using</s>as the start token like the summerization models it again gave weird results.['</s><s> created Python in 1991?']So to me it seems thatadjust_logits_during_generationand</s>as thedecoder_start_tokenis only working for the pre-trained summrization models and not for other bart-large or bart-base finetuned models.@chrisdoyleIEcan you also confirm this with your modelfirst do beam search usingeos_token_idas thedecoder_start_token_iduse the bos_token as start token (as usual) and return logits as it is fromadjust_logits_during_generationinstead of forcing<s>. You can do it usingdef adjust_logits(logits, **kwargs):
    return logits
model.adjust_logits_during_generation = adjust_logits@sshleifer, if</s>is thedecoder_start_tokenthen why don’t we set it when fine-tuning the model ? Also in fairseq they seem to use the bos token<s>as the first prefix.HereSorry for this long issue, but no other way to explain it.
2020-08-11T10:31:04Z
@valhallathis is a stellar investigation, thank you very much!  I’ll need about 2 weeks to submit my thesis then look forward to taking a look.
2020-08-11T12:35:45Z
@sshleifer,@chrisdoyleIEAfter modifying theadjust_logits_during_generationBLUE-4 score for my model went from 13.09 to 19.14. So this indeed is an issue.
2020-08-11T19:09:35Z
Can you PR a fix/ or update the docs with the suggested behavior? I am a little busy at the moment.
2020-08-16T19:08:03Z
valhalla:After modifying theadjust_logits_during_generationBLUE-4 score for my model went from 13.09 to 19.14. So this indeed is an issue.In the config of  your finetuned model, what was decoder_start_token_id?In your training code, what did a batch “look like”?
2020-08-17T04:50:18Z
decoder_start_token_idwas not defined in theconfig. Sogenerateusedboswhen it’s not defined in theconfig. And the batches were simply prepared like thissrc_text = "Python was created by <hl> Guido van Rossum <hl> in 1991."
target _text = "Who created python ?"
input_ids = tokenizer(src_text, return_tensors="pt")["input_ids"]
target_ids =  tokenizer(target , return_tensors="pt")["input_ids"]
decoder_input_ids = target_ids[:, :-1].contiguous()
labels = target_ids[:, 1:].clone()