code: def attention_lm_moe_large():
    
    hparams = attention_lm_moe_base()
    hparams.num_hidden_layers = 5
    hparams.moe_layers = "3"
    hparams.hidden_size = 1024
    hparams.num_heads = 16
    hparams.filter_size = 4096
    hparams.moe_hidden_sizes = "4096"
    hparams.moe_num_experts = 128
    hparams.layer_prepostprocess_dropout = 0.2
    return hparams
docs: Large model for distributed training.
  
    Over 1B parameters, so requires multi-gpu training due to memory
     requirements.
  
    on lm1b_32k:
       After 45K steps on 8 GPUs (synchronous):
          eval_log_ppl_per_token = 3.18
          eval_ppl_per_word = exp(1.107893 * eval_log_ppl_per_token) = 33.9
  
    Returns:
      an hparams object.