code: def xmoe_top_2():
    
    hparams = xmoe_dense_4k()
    moe.set_default_moe_hparams(hparams)
    hparams.mesh_shape = "all:8"
    hparams.layout = "batch:all;experts:all"
    return hparams
docs: Mixture of experts (16 experts).