code: def transformer_base_range(rhp):
    
    # After starting from base, set intervals for some parameters.
    rhp.set_float("learning_rate", 0.3, 3.0, scale=rhp.LOG_SCALE)
    rhp.set_discrete("learning_rate_warmup_steps",
                     [1000, 2000, 4000, 8000, 16000])
    rhp.set_float("initializer_gain", 0.5, 2.0)
    rhp.set_float("optimizer_adam_beta1", 0.85, 0.95)
    rhp.set_float("optimizer_adam_beta2", 0.97, 0.99)
    rhp.set_float("weight_decay", 0.0, 1e-4)
docs: Small range of hyperparameters.