Resume in train_unconditional.py
I’m just running some initial tests/experiments intrain_unconditional.py, to get a sense of how my data/images are going to work with Diffusers and I’d like to continue training from a saved checkpoint. I modified the script to use:if args.overwrite_output_dir:
        model = UNet2DModel(
            // original model code
        )
    else:
        model_dir = args.output_dir + '/unet/'
        model = UNet2DModel.from_pretrained(model_dir)which does appear to successfully load my checkpoint (based on the initial loss). However, when I run it, I see that the loss is clearly diverging. I’m guessing I don’t have the correct scheduler/optimizer settings, but I’m not sure how to load those.Any tips appreciated.
2022-11-11T08:48:56Z
Hi@jbmaxwell! I think that’s an interesting use case, would you mindopening an issue in the repo?Thanks a lot!
2022-11-11T16:05:28Z
Okay, will do.
2022-11-11T16:06:55Z
Thank you!
2022-11-11T23:59:57Z
Actually, it looks like the problem is really just that the learning rate doesn’t continue from where it was at the end of the last run—i.e., it just runs the whole lr schedule again, as if training from scratch. So it starts okay, diverges for a while, then starts to converge again.Probably it’s something I’ve missed… ??