DiffusionPipeline.from_pretrained SDXL model: Input type and bias type should be the same
I am trying to run DiffusionPipeline on custom SDXL model and am having trouble with input and bias type. Anyone else have this issue/find a way around?Code:from diffusers import DiffusionPipeline
import torch
model = "stabilityai/stable-diffusion-xl-base-1.0"
pipe = DiffusionPipeline.from_pretrained(model, torch_dtype = torch.float16,
)
pipe.to("cuda")
pipe.load_lora_weights("/mys3bucket/model/", weight_name = "pytorch_lora_weights.safetensors"
)

prompt = ["photo of (zwx child) in outer space, nasa suit, space station, RAW candid cinema, 16mm, color graded portra 400 film, remarkable color, ultra realistic, textured skin, remarkable detailed pupils, shot with cinematic camera"]
negative_prompt = "lowres"
generator = torch.Generator("cuda")

for seed in range(1):

image = pipe(prompt = prompt[seed], negative_prompt=negative_prompt, generator=generator, num_inference_steps=25)
image = image.images[0]
image.save(f"/mys3bucket/output_images/{seed}.png")Error:Traceback (most recent call last):
  File "/home/ubuntu/sdxl2.py", line 25, in <module>
    image = pipe(prompt = prompt[seed], negative_prompt=negative_prompt, generator=generator, num_inference_steps=25)
  File "/usr/local/lib/python3.10/dist-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py", line 847, in __call__
    image = self.vae.decode(latents / self.vae.config.scaling_factor, return_dict=False)[0]
  File "/usr/local/lib/python3.10/dist-packages/diffusers/utils/accelerate_utils.py", line 46, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/diffusers/models/autoencoder_kl.py", line 270, in decode
    decoded = self._decode(z).sample
  File "/usr/local/lib/python3.10/dist-packages/diffusers/models/autoencoder_kl.py", line 256, in _decode
    z = self.post_quant_conv(z)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/conv.py", line 463, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/conv.py", line 459, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Input type (c10::Half) and bias type (float) should be the sameMore details here:github.com/huggingface/diffusersDiffusionPipeline.from_pretrained SDXL model: Input type and bias type should be the sameopened02:43AM - 05 Aug 23 UTCclosed03:26PM - 05 Aug 23 UTCcaldwecgbug### Describe the bug

I am running into an error when generating images. I tri…ed the below fix and it only let me generate the first image, subsequent images would fail to the error below. 

for module in pipe.vae.modules():
    if isinstance(module, torch.nn.Conv2d):
        module.half()

### Reproduction

Here is the code I am using:

`
from diffusers import DiffusionPipeline
import torch
model = "stabilityai/stable-diffusion-xl-base-1.0"
pipe = DiffusionPipeline.from_pretrained(model, torch_dtype = torch.float16,
        )
pipe.to("cuda")
pipe.load_lora_weights("/mys3bucket/model/", weight_name = "pytorch_lora_weights.safetensors"
        )

prompt = ["photo of (zwx child) in outer space, nasa suit, space station, RAW candid cinema, 16mm, color graded portra 400 film, remarkable color, ultra realistic, textured skin, remarkable detailed pupils, shot with cinematic camera"]
negative_prompt = "lowres"
generator = torch.Generator("cuda")

for seed in range(1):

    image = pipe(prompt = prompt[seed], negative_prompt=negative_prompt, generator=generator, num_inference_steps=25)
    image = image.images[0]
    image.save(f"/mys3bucket/output_images/{seed}.png")
`



### Logs

```shell
Traceback (most recent call last):
  File "/home/ubuntu/sdxl2.py", line 25, in <module>
    image = pipe(prompt = prompt[seed], negative_prompt=negative_prompt, generator=generator, num_inference_steps=25)
  File "/usr/local/lib/python3.10/dist-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py", line 847, in __call__
    image = self.vae.decode(latents / self.vae.config.scaling_factor, return_dict=False)[0]
  File "/usr/local/lib/python3.10/dist-packages/diffusers/utils/accelerate_utils.py", line 46, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/diffusers/models/autoencoder_kl.py", line 270, in decode
    decoded = self._decode(z).sample
  File "/usr/local/lib/python3.10/dist-packages/diffusers/models/autoencoder_kl.py", line 256, in _decode
    z = self.post_quant_conv(z)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/conv.py", line 463, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/conv.py", line 459, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Input type (c10::Half) and bias type (float) should be the same
```


### System Info

Running on AWS g4dn.xlarge instance

Torch + cuda version: 1.13.1+cu116
Nvidia Version: NVIDIA-SMI 510.108.03   Driver Version: 510.108.03   CUDA Version: 11.6  


### Who can help?

@william @patrickvonplaten, and @sayakpaul
2023-08-06T01:46:25Z
Solved by updating transformers/xformers/torchvision/accelerators