code: def transformer_moe_8k():
    
    hparams = transformer_moe_base()
  
    hparams.batch_size = 8192
    hparams.max_length = 0  # max_length == batch_size
    hparams.eval_drop_long_sequences = True
    hparams.min_length_bucket = 256  # Avoid cyclic problems for big batches
  
    hparams.default_ff = "sep"
    hparams.hidden_size = 1024
  
    return hparams
docs: Hyper parameters specifics for long sequence generation.