How to train causal language model
I found an example in [transformers/examples/pytorch/language-modeling/run_clm.py]According to the script, the trainer use dafault_data_collator to model causal language modelling.(transformers/examples/pytorch/language-modeling/run_clm.py at 98dda8ed03ac3f4af5733bdddaa1dab6a81e15c1 · huggingface/transformers · GitHub)Shouldn’t we use DataCollatorForLanguageModeling to shift input and output by 1 token instead? It seems that dafault_data_collator can’t achieve this goal.