code: def attention_lm_moe_24b_diet():
    
    hparams = attention_lm_moe_large_diet()
    hparams.moe_hidden_sizes = "12288"
    hparams.moe_num_experts = 1024
    hparams.batch_size = 4096
    return hparams
docs: Unnecessarily large model with 24B params - because we can.