How to resume training from checkpoint
i want to resume training unconditional image generation from latest checkpoint. But after loaded checkpoints, doesn’t continue training. Here is my code and output:!accelerate launch --multi_gpu train_unconditional.py–dataset_name=“-------”–resolution=64 --center_crop --random_flip–output_dir=“--------”–train_batch_size=16–num_epochs=1000–gradient_accumulation_steps=1–use_ema–learning_rate=1e-4–lr_warmup_steps=500–mixed_precision=no–push_to_hub–resume_from_checkpoint=“checkpoint-4000”Output:Resuming from checkpoint checkpoint-400004/08/2024 08:51:40 - INFO - accelerate.accelerator - Loading states from checkpoint-4000The config attributes {‘decay’: 0.9999, ‘inv_gamma’: 1.0, ‘min_decay’: 0.0, ‘optimization_step’: 4000, ‘power’: 0.75, ‘update_after_step’: 0, ‘use_ema_warmup’: True} were passed to UNet2DModel, but are not expected and will be ignored. Please verify your config.json configuration file.04/08/2024 08:51:41 - INFO - accelerate.checkpointing - All model weights loaded successfully04/08/2024 08:51:42 - INFO - accelerate.checkpointing - All optimizer states loaded successfully04/08/2024 08:51:42 - INFO - accelerate.checkpointing - All scheduler states loaded successfully04/08/2024 08:51:42 - INFO - accelerate.checkpointing - All dataloader sampler states loaded successfully04/08/2024 08:51:42 - INFO - accelerate.checkpointing - All random states loaded successfully04/08/2024 08:51:42 - INFO - accelerate.accelerator - Loading in 0 custom statesthis is output and stopped runningHelp me pls ))