4 bytes * number of parameters for fp32 training
6 bytes * number of parameters for mixed precision training (maintains a model in fp32 and one in fp16 in memory)

Optimizer States:

8 bytes * number of parameters for normal AdamW (maintains 2 states)
2 bytes * number of parameters for 8-bit AdamW optimizers like bitsandbytes
4 bytes * number of parameters for optimizers like SGD with momentum (maintains only 1 state)

Gradients