Customizing generation config for Trainer's training loop evaluation
When using Trainer’s training loop, I’d like to be able to customize the parameters for generation (used for evaluation). I tried the following approach:gen_config = copy.deepcopy(model.generation_config)
gen_config.update(
    max_new_tokens=10,
    early_stopping=False,
    num_beams=1,
    no_repeat_ngram_size=0,
)
gen_config.validate()
training_args = Seq2SeqTrainingArguments(
    ....,
    predict_with_generate=True,
    generation_config=gen_config,
)
trainer = Seq2SeqTrainer(
    model=model,
    args=training_args,
    eval_dataset=...,
    ....
)
trainer.train()However, the values are not used (for example,max_new_tokensis not respected). This warning is generated:Some non-default generation parameters are set in the model config. These should go into a GenerationConfig file (Text generation strategies) instead. This warning will be raised to an exception in v4.41.Non-default generation parameters: {‘early_stopping’: True, ‘num_beams’: 4, ‘no_repeat_ngram_size’: 3, ‘forced_bos_token_id’: 0, ‘forced_eos_token_id’: 2}along with:/opt/conda/lib/python3.10/site-packages/transformers/generation/utils.py:1128: UserWarning: Using the model-agnostic defaultmax_length(=20) to control the generation length. We recommend settingmax_new_tokensto control the maximum length of the generation.I tried to follow the advice from the first warning by saving the modified configuration into a file and then loading it:gen_config = copy.deepcopy(model.generation_config)
gen_config.update(
    max_new_tokens=10,
    early_stopping=False,
    num_beams=1,
    no_repeat_ngram_size=0,
)
gen_config_file = "gen_config.json"
gen_config.to_json_file(gen_config_file)
training_args = Seq2SeqTrainingArguments(
    ....,
    predict_with_generate=True,
    generation_config=gen_config_file,
)
...but the warning stays the same.How can I customize the evaluation generation when using Trainer? Thank you for any advice.
2024-01-31T09:44:45Z
I found the problem. The warning was actually triggered by Trainer saving the model viasave_pretrained, not by generation. I submitted an issue:BART-base save_pretrained triggers a warning about GenerationConfig · Issue #28793 · huggingface/transformers · GitHub