code: def transformer_tall_pretrain_lm():
    
    hparams = transformer_tall()
    hparams.learning_rate_constant = 2e-4
    hparams.learning_rate_schedule = ("linear_warmup*constant*cosdecay")
    hparams.optimizer = "adam_w"
    hparams.optimizer_adam_beta1 = 0.9
    hparams.optimizer_adam_beta2 = 0.999
    hparams.optimizer_adam_epsilon = 1e-8
    # Set max examples to something big when pretraining only the LM, definitely
    # something an order of magnitude bigger than number of train steps.
    hparams.multiproblem_schedule_max_examples = 5e8
    # Set train steps to learning_rate_decay_steps or less
    hparams.learning_rate_decay_steps = 5000000
    return hparams
docs: Hparams for transformer on LM pretraining (with 64k vocab).