Is the trainer's seed reset at every model_init?
Becauseseedis a TrainingArgument, we can use it as in hyperparameter optimization if we’d want to, e.g.def hparams_ray(trial):
    from ray import tune

    return {
        "learning_rate": tune.loguniform(1e-6, 1e-3),
        "per_device_train_batch_size": tune.choice([4, 8, 16, 32]),
         "seed": tune.choice(range(1, 43)),
    }But I am looking for clarification what happens when we do not includeseedhere. Is the seed (or any training argument for that matter), reset at each trial? This is important: if it is not reset, then every trial has a different starting seed and hyperparameter search is not independent.
2022-03-28T14:22:31Z
Yes, it’s done atthat linespecifically.
2022-03-28T15:08:49Z
I guess I misunderstand how Ray (or Optuna) interacts with the trainer then.I thought that the Trainer is never copied/reinitialized but only the model is reinitialized for every trial. You seem to suggest that a new trainer is created for every trial (and in the trainer init, set_seed is called), is that correct?
2022-03-28T16:52:59Z
Pasted the wrong link, sorry. It is reinitialized (with the seed set) at each new call totrainhere
2022-03-28T16:56:47Z
Great, that was what I was looking for. Thanks!