Gradients

4 bytes * number of parameters for either fp32 or mixed precision training (gradients are always kept in fp32)

Forward Activations

size depends on many factors, the key ones being sequence length, hidden size and batch size.