Bart-base rouge scores
Has anyone finetunedbart-baseon xsum or cnn summarization task and willing to report the rouge score they got?I just got 15.5 for xum which feels low, since bart-large can get to 22 ish.@astariul@valhalla@VictorSanh?
2020-08-12T05:44:41Z
@sshleifer, could it be due to theadjust_logitsissue ? Just a guess but as I posted there, after modifying theadjust_logits_during_generationBLUE-4 score for my model went from 13.09 to 19.14 forbart-base
2020-08-14T14:34:55Z
@sshleifercould you also try usingbosasdecoder_start_token_idand modifyingadjust_logits_during_generationto return logits as is instead of forcingbos? If you also get bump in ROUGE score we can confirm the issue. Thanks !
2020-08-15T18:00:41Z
Possible suggestion that saves on the re-training could be to check the perplexity values and compare to paper
2020-08-25T15:46:01Z
I got 16.6 ROUGE 2 on XSUM, in 3 epochs/ 6hrs
2020-08-25T16:24:38Z
bart-base doesn’t seem to be good then, in my other seq2seq experiment t5-small performed similar/better to bart-base
2020-09-05T18:22:20Z
Made a google doc to aggregate experiment results. Please add any interesting results!docs.google.comSeq2Seq Finetuning LeaderboardPlease state what you did and what you got   Pegasus-large on xsum: 	Released model: 46.87/24.46/39.15  (Rouge1/Rouge2/RougeL) finetuned:  {"rouge1": 46.8248, "rouge2": 23.9987, "rougeL": 38.6751, "n_obs": 11333, "runtime": 4228.170863628387,...
2020-10-27T12:08:43Z
How can I change the adjust_logits_during_generation ? thanks
2020-10-27T14:33:43Z
By editing the code!
2020-10-27T15:26:26Z
Can you provide a example ? I saw the source code ofadjust_logits_during_generationand it directly returns the logits.
2020-10-27T16:32:19Z
github.comhuggingface/transformers/blob/master/src/transformers/modeling_bart.py#L1100):return {"input_ids": None,  # encoder_outputs is defined. input_ids not needed"encoder_outputs": encoder_outputs,"past_key_values": past,"decoder_input_ids": decoder_input_ids,"attention_mask": attention_mask,"use_cache": use_cache,  # change this to avoid caching (presumably for debugging)}def adjust_logits_during_generation(self, logits, cur_len, max_length):if cur_len == 1 and self.config.force_bos_token_to_be_generated:self._force_token_id_to_be_generated(logits, self.config.bos_token_id)elif cur_len == max_length - 1 and self.config.eos_token_id is not None:self._force_token_id_to_be_generated(logits, self.config.eos_token_id)return logits@staticmethoddef _force_token_id_to_be_generated(scores, token_id) -> None:"""force one of token_ids to be generated by setting prob of all other tokens to 0 (logprob=-float("inf"))"""scores[:, [x for x in range(scores.shape[1]) if x != token_id]] = -float("inf")in the futuregit grep adjust_logits_during_generation
2020-10-27T16:56:12Z
thanks