Running into out of memory issues
Hi,I’m currently trying to train huggingface Diffusers for 2D image generation task with images as input.Training on AWS G5 instances i.e., A10G GPU’s with 24GB GPU memory.I’m running into out of memory issues when I go beyond image size of 256x256 and batch size of 8.Results with Image size = 256 and batch size = 8 is unacceptableI did use gradient accumulation and mixed precision training.Using 1 attention block only.Trying to understand is it a genuine memory issue or can it be solved by some other approaches?Is diffusion models so heavy that even a 24GB memory GPU is insufficient?What’s the typical memory requirement for running image to image diffusion models, for generating images resolution higher than 512x512Thanks and regardsKVS Moudgalya
2023-04-06T03:13:27Z
Hey, sorry that you’re running into the issues.The first thing I would try doing is to usexFormers:diffusers/examples/text_to_image at main · huggingface/diffusers · GitHub
2023-04-24T11:30:50Z
Thanks@sayakpaulfor your response.I did enable xFormers in the huggingface diffusers, there is slight a bit of improvement in computational speed, but couldn’t not much in terms of memory, can’t increase image size above 256 and batch size above 8, getting out of memory issues if I go beyond these dimensions.