GPU memory calculator
I am looking for a calculator tool that can estimate the amount of memory a GPU machine instance will use in advance. Specifically, I’m interested in the following configuration:FSDP Zero 2Model: e.g., LLama3 8B-LoraMax tokens: 8K16-bit precisionGradient checkpointingNo quantizationAny guidance or recommendations would be greatly appreciated! Thank you.Edit: I know about hf tool (Understanding how big of a model can fit on your machine) but this is rudimentary and lacks many of parameters such as max sequence token, lora, …
2024-07-04T11:34:41Z
This one has a bit more than the HF one:Tech 42. There’s an option to add a custom model.
2024-07-05T12:33:48Z
You can try the Python code for the calculations. Use the formulas with all parameters:total_memory=(model_memory+activation_memory+lora_memory)Xcheckpointing_factorXfsdp_factor