How to use lora with SDXL img2img?
I am trying to apply a lora to the SDXL refiner img2img pipeline. I’ve tried multiple sdxl loras that work with the base model and pipeline but when i try them with StableDiffusionXLImg2ImgPipeline and the refiner model it errors (I have setlow_cpu_mem_usage=Falseandignore_mismatched_sizes=Trueto no avail)StableDiffusionXLImg2ImgPipelinehasload_lora_weightsso I’m assuming this should work.Is this possible and if so what am I missing?{'lora': 'Norod78/SDXL-Caricaturized-Lora', 'subfolder': None, 'weight_name': 'SDXL-Caricaturized-Lora.safetensors'}
It might be incompatible with stabilityai/stable-diffusion-xl-refiner-1.0
Cannot load because down.weight expected shape tensor(..., device='meta', size=(8, 1536)), but got 
torch.Size([8, 1280]). If you want to instead overwrite randomly initialized weights, please make sure to 
pass both `low_cpu_mem_usage=False` and `ignore_mismatched_sizes=True`. 
For more information, see also: https://github.com/huggingface/diffusers/issues/1619#issuecomment-1345604389 as an example.
2023-09-19T23:25:24Z
I am having the same issueit’s not clear if you have to reload also lora and textual inversion embedding on the refiner if you use the base model to init the refiner pipeline to mehave you found a solution?I tried to load them in different order but nothing works
2023-09-24T16:12:15Z
I’m having the same issue. Is there any example that you’ve found of people doing this successfully?
2023-09-29T00:18:57Z
I don’t think you can use a Lora with the refiner. Using the base modelstabilityai/stable-diffusion-xl-base-1.0andStableDiffusionXLImg2ImgPipelineworks. And then run it through the refiner without the lora.