

There are the input and output that are being passed and returned by the forward and the backward functions and the 
forward activations saved for gradient computation.
Temporary Memory
Additionally, there are all kinds of temporary variables which get released once the calculation is done, but in the 
moment these could require additional memory and could push to OOM. Therefore, when coding it's crucial to think 
strategically about such temporary variables and sometimes to explicitly free those as soon as they are no longer needed.
Functionality-specific memory
Then, your software could have special memory needs. For example, when generating text using beam search, the software 
needs to maintain multiple copies of inputs and outputs.
forward vs backward Execution Speed
For convolutions and linear layers there are 2x flops in the backward compared to the forward, which generally translates 
into ~2x slower (sometimes more, because sizes in the backward tend to be more awkward). Activations are usually 
bandwidth-limited, and it’s typical for an activation to have to read more data in the backward than in the forward 
(e.g. activation forward reads once, writes once, activation backward reads twice, gradOutput and output of the forward, 
and writes once, gradInput).
As you can see, there are potentially a few places where we could save GPU memory or speed up operations. 
Now that you understand what affects GPU utilization and computation speed, refer to 
the Methods and tools for efficient training on a single GPU documentation page to learn about 
performance optimization techniques. 