code: def attention_lm_moe_base_ae():
    
    hparams = attention_lm_moe_base_long_seq()
    hparams.attention_type = AttentionType.LOCAL_EXPERTS
  
    hparams.learning_rate = 0.05
    hparams.learning_rate_warmup_steps = 10000
    # According to noam, ("n", "da") seems better for harder-to-learn models
    # hparams.layer_preprocess_sequence = "n"
    # hparams.layer_postprocess_sequence = "da"
    return hparams
docs: Base model with attention expert.