code: def lmx_base():
    
    hparams = transformer.transformer_tpu()
    # sharing is counterproductive when underparameterized
    hparams.shared_embedding_and_softmax_weights = False
    # we judge by log-ppl, so label smoothing hurts.
    hparams.label_smoothing = 0.0
    # This makes the batch size on GPU the same as on TPU for a packed problem
    # with sequence length 256.
    # TODO(noam): fix the mess that is the data reading pipeline.
    hparams.max_length = 256
    # larger batch since we only have a decoder
    hparams.batch_size = 4096
    # save some memory so we can have a larger model
    hparams.activation_dtype = "bfloat16"
    return hparams
docs: Transformer on languagemodel_lm1b32k_packed.  50M Params.