New seq2seq tool: search hparam space with run_eval.py
FYI, there is a new tool available to you - you can now search the hparam space withrun_eval.py.It’s calledrun_eval_search.pyIt uses the same arguments asrun_eval.py, but allows you to parametrize the hparams, so in addition to the normal args you can pass:--search="num_beams=8:11:15 length_penalty=0.9:1.0:1.1 early_stopping=true:false"and it’ll search all the possible combinations and at the end print a table of results sorted by the scores of the task. e.g.:bleu  | num_beams | length_penalty | early_stopping
----- | --------- | -------------- | --------------
41.35 |        11 |            1.1 |              0
41.33 |        11 |            1.0 |              0
41.33 |        11 |            1.1 |              1
41.32 |        15 |            1.1 |              0
41.29 |        15 |            1.1 |              1
41.28 |        15 |            1.0 |              0
41.25 |         8 |            1.1 |              0
41.24 |        11 |            1.0 |              1
41.23 |        11 |            0.9 |              0
41.20 |        15 |            1.0 |              1
41.18 |         8 |            1.0 |              0You can have one or more params searched.Here is an example of a full command:PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval_search.py \
facebook/wmt19-$PAIR $DATA_DIR/val.source $SAVE_DIR/test_translations.txt \
--reference_path $DATA_DIR/val.target --score_path $SAVE_DIR/test_bleu.json \
--bs $BS --task translation \
--search="num_beams=1:5 length_penalty=0.9:1.1 early_stopping=true:false"If you encounter any issues please let me know.It’s documented here:https://github.com/huggingface/transformers/blob/master/examples/seq2seq/README.md#run_eval-tips-and-tricks.@sshleiferand I added some more goodies inrun_eval.py- you will find them all documented at that url.Enjoy.p.s. edited to remove things that are going to change based on Sam’s comment below.
2020-09-16T22:07:28Z
Great work!There are only two possible sets of keys to get fromrun_eval.pysincescore_fn = calculate_bleu_score if "translation" in args.task else calculate_rougeYou shouldn’t hard code the possible tasks any more than that IMO.
2020-09-16T22:25:01Z
ah, thank you for clarifying that - I will adjust it to follow the same logic.
2020-09-17T06:04:14Z
This is awesome ! Thanks@stas
2020-09-17T11:31:32Z
I haven’t checked the code, I’m on mobile now. But are there many scenarios where we actually need to do hyperparameters search on theevaluation/inference side?  In addition, does this use the optuna implementation that is being worked on in the trainer by@sgugger, or is it a separate implementation?
2020-09-17T12:11:00Z
When you train a seq2seq model on new summ or translation dataset or other seq2seq task and want to decide how many beams to use, should use length penalty or not, what should be the max seq length, what should be theno_repeat_ngram_sizeetc, all of these parameter affect the metrics , so this tool helps to make those decisions,It does not useoptuna,it just usesitetools.productto enumerate the different combinations and evaluate on them