BART - Input format
Hi,Due to recent code changes by@sshleifer, I am trying to understand what is desired for BART’s input for training and generation, and whether the codebase is reflecting it properly as I’ve encountered some inconsistencies.I am assuming bothsrc_idsandtgt_idsare encoded with a BART tokenizer, and therefore have the format of [bos, token1, token2, …, eos].Looking attransformers/examples/seq2seq/finetune.py#L151decoder_input_ids = shift_tokens_right(tgt_ids)means thateoswill be the first token andboswill be the second token.This has an effect on generation:We needdecoder_start_token_id=eos_token_id.The first actually generated token (i.e. afterdecoder_start_token_id) will bebos.Questions:The default value fordecoder_start_token_idis missing fromfacebook/bart-baseandfacebook/bart-large-mnli, which means it falls back tobos. The other BART models haveeosas theirdecoder_start_token_id. Why is the difference? Looks to me that using finetune.py withbart-base/bart-large-mnliwill not have generation as intended.In fairseq’s implementation the equivalent fordecoder_start_token_idis set tobos:fairseq/models/bart/hub_interface.py#L123. Can you please explain why did you decide to use the format of[eos, bos, token1, token2, ...]fordecoder_input_idsinstead of[bos, token1, token2, ...]?Is there still need forforce_bos_token_to_be_generated?It was introduced in transformers/pull/6526 (new user, can’t add another link), when the first token ofdecoder_input_idswasbosand the second was the first regular token of the target sequence (transformers/examples/seq2seq/finetune.py#L144). Using it now shouldn’t have any effect, if I understand correctly (because a trained model will easily learn to always outputbosin this position anyway).Thanks!
2020-09-11T15:02:36Z
This is agreatquestion and super confusing.You can get a good snapshot of my understandinghere. I don’t think that will clear everything up, but if you could read that and let me know what you still don’t understand it would be helpful.If I encounter empirical evidence that I should changedecoder_start_token_idfor any model I will do so.
2023-12-11T04:00:01Z
I hope you folks don’t mind me reviving this old topic, but it seems relevant to my issues with BOS and EOS.I am running different training experiments with different input formats for a facebook/bart-base model augmented by unlimiformer (retrieval augmented to store larger documents).The standard format is a Long Document (LD): a single string.The second format is a Knowledge Graph of the LD: a single string, but with each relation in the graph entered as a sequence"<s> rel1 </s><s> rel2 </s> ... <s> reln </s>"The target is a summary: a single string.The odd thing is that, in my initial conda environment, due to constraints suggested by the unlimiformer authors, I ended up with transformers 4.34.1. Somehow this got corrupted and I built a new environment and since the constraints seemed unsatisfiable, I removed all and ended up with transformers 4.35.2.The difference was stark.With 4.34.1, LDs result in summaries of length 70-130. In contrast KGs result in summaries of length 600-1000: with lots of lists. Moreover, although the KG inputs are about one-fifth the length, the summaries took much longer to generate.With 4.35.2, both summaries were very similar in length and training is twice as fast (12 hours to get to 21k steps).Also, not sure what role themax_length=128andgeneration_max_length=1024parameter play here as results seem to violate themax_lengthconstraint.If you feel this is unrelated, I’m happy to open a new post.
2023-12-13T01:23:16Z
I found that theadd_special_tokens=Falsesetting seems to be driving the above behaviour. And this is dependent on conda environment (for a fixed Transformer version 1.34.1).
2023-12-13T01:31:08Z
Hey, has anyone got any answers to the above? Thanks!