code: def lmx_moe_h1k_f8k_x16():
    
    hparams = lmx_h1k_f4k()
    hparams.filter_size = 8192
    hparams.ffn_layer = "local_moe_tpu"
    hparams.moe_num_experts = 16
    hparams.weight_dtype = "bfloat16"
    hparams.batch_size = 8192
    return hparams
docs: Transformer with mixture of experts.  890M Params.