Hyper params search for model config
Hello everyoneI am currently training the t5 model for the seq2seq task right now, and I wonder if we can use Optuna to do the hyper-params search for the config likenum_layersORnum_heads.According to Mr. Sgugger in this discussionThe hyperparams you can tune must be in theTrainingArgumentsyou passed to your Trainer. If you have custom ones that are not inTrainingArguments, just subclassTrainingArgumentsand add them in your subclass.Thehp_spacefunction indicates the hyperparameter search space (see the code of the default for optuna or Ray intraining_utils.pyand adapt it to your needs) and thecompute_objectivefunction should return the objective to minize/maximize.But it was almost 3 years ago, I am wondering there is any method to achieve this now??Becausenum_layersornum_headsare not in the args of Trainer, I really don’t know how to search for this.