code: def attention_lm_moe_small():
    
    hparams = attention_lm_moe_base()
    hparams.num_hidden_layers = 4
    hparams.hidden_size = 512
    hparams.filter_size = 2048
    hparams.moe_num_experts = 128
    hparams.moe_layers = "2"
    return hparams
docs: Cheap model for single-gpu training.
  
    on lm1b_32k:
       ~312M params
       1.6 steps/sec on  [GeForce GTX TITAN X]
       After 50K steps on 8 GPUs (synchronous):
          eval_log_ppl_per_token = 3.31
  
    Returns:
      an hparams object.