Duplicate hyperparameter tuning trials with Ray and `hyperparameter_search`
I’m usinghyperparameter_searchfor hyperparamter tuning with the following configurations:def tune_config_ray(trial):
        return {"learning_rate": tune.choice([5e-5, 4e-5, 3e-5, 2e-5]),
                "num_train_epochs": tune.choice([4]),
                "per_device_train_batch_size": tune.choice([16])
               }

    best_trial = trainer.hyperparameter_search(hp_space=tune_config_ray,
                                               backend='ray',
                                               direction='maximize',
                                               n_trials=4,
                                               )Based on the config, there are 4 unique combinations for thelearning_rate,num_train_epochs, andper_device_train_batch_size. However, when I run the tuning (as you can see below), I see some duplicates among the trials. I wonder why this happens and how I can have non-duplicate trials?  Is this possibly because ray is also tuning some other hyperparameters that are not listed in the report and that is why I see some duplicates?+------------------------+----------+-------+-----------------+--------------------+-------------------------------+
| Trial name             | status   | loc   |   learning_rate |   num_train_epochs |   per_device_train_batch_size |
|------------------------+----------+-------+-----------------+--------------------+-------------------------------|
| _objective_7efa7_00000 | RUNNING  |       |           3e-05 |                  4 |                            16 |
| _objective_7efa7_00001 | PENDING  |       |           2e-05 |                  4 |                            16 |
| _objective_7efa7_00002 | PENDING  |       |           5e-05 |                  4 |                            16 |
| _objective_7efa7_00003 | PENDING  |       |           3e-05 |                  4 |                            16 |
+------------------------+----------+-------+-----------------+--------------------+-------------------------------+
2021-08-03T06:32:04Z
cc@richardliaw
2021-08-03T07:21:07Z
@sguggerthanks for the ping@phosseiniunfortunately you can’t do a sample without replacement with Ray Tune right now, but you can however easily do a grid search:def tune_config_ray(trial):
        return {"learning_rate": tune.grid_search([5e-5, 4e-5, 3e-5, 2e-5]),
                "num_train_epochs": tune.choice([4]),
                "per_device_train_batch_size": tune.choice([16])
               }

    best_trial = trainer.hyperparameter_search(hp_space=tune_config_ray,
                                               backend='ray',
                                               direction='maximize',
                                               n_trials=1,  # you may need to set this to 1, or else you'll evaluate the grid search many times over.
                                               )