[Ray] How to get the best model per trial
Hi everyone,I’m trying to use the feature “hyperparameter_search” but cannot get the best model per the trial. Each trial always get the last value.I see the code line here calling the Ray tune function to get the best model without the “scope” parameter passed. Default value of that function is “last”.github.comhuggingface/transformers/blob/123cce6ffcafe6ecece7eb9914395d2ebcc98a48/src/transformers/integrations.py#L294# special attr set by tune.with_parametersif hasattr(trainable, "__mixins__"):dynamic_modules_import_trainable.__mixins__ = trainable.__mixins__analysis = ray.tune.run(dynamic_modules_import_trainable,config=trainer.hp_space(None),num_samples=n_trials,**kwargs,)best_trial = analysis.get_best_trial(metric="objective", mode=direction[:3])best_run = BestRun(best_trial.trial_id, best_trial.last_result["objective"], best_trial.config)if _tb_writer is not None:trainer.add_callback(_tb_writer)return best_rundef run_hp_search_sigopt(trainer, n_trials: int, direction: str, **kwargs) -> BestRun:from sigopt import ConnectionIs there any way to workaround this?Thanks!
2021-11-18T11:10:21Z
Thanks for raising this.I am having the same problem with the Optuna backend.