code: def attention_lm_attention_moe_tiny():
    
    hparams = attention_lm_moe_small()
    hparams.moe_layers = ""
    hparams.attention_num_experts = 128
    hparams.filter_size = 8192
    hparams.attention_type = AttentionType.LOCAL_EXPERTS
    return hparams
docs: Cheap model for debugging.
  
    Returns:
      an hparams object.