Stable diffusion bs>1 uses negative as prompt
Using negative prompts with a batch size > 1 ([prompt]*x , [negative]*x in their respective ways) causes the image to be more towards the negative prompt than the normal prompt.pipe([prompt]*x, negative_prompt=[negative]*x, width=width, height=height, guidance_scale=guidance_scale, num_inference_steps=steps)I can link the notebook I’m using if necessary but the code here is so simple that I feel like I don’t need to.(seed 23335, w/h = 512, prompt = “frog”, negative = “bird”, x = 2, guidance_scale = 7.5, steps = 150, using thelms = LMSDiscreteScheduler(     beta_start=0.00085,      beta_end=0.012,      beta_schedule="scaled_linear" ))image1547×706 195 KBThis might fit better as an issue on the diffusers github but I really don’t know how to use that feature.that first is using waifu-diffusion v1.3 converting with the conversion script in the repo, but it works the same with the normal stable diffusion v1-4 model too: (linked image so that the “1 embed per new user” thing doesn’t yell at me)https://media.discordapp.net/attachments/1024588665596411944/1028159556561420318/unknown.png?width=1258&height=557edit: this is fixed by passing both as a single string, not using x or using it as a list, but passing “num_images_per_prompt=x” when calling it, leaving this here in case other people find the same problem (same settings just using num_images_per_prompt=2 instead of making the prompt a list and multiplying by 2:https://media.discordapp.net/attachments/1024588665596411944/1028162978689847396/unknown.png)
2022-10-08T11:41:30Z
crumb:edit: this is fixed by passing both as a single string, not using x or using it as a list, but passing “num_images_per_prompt=x” when calling itI was going to suggest that, glad you found it.However, the previous method should also work, I’ll take a look and report back.Thanks a lot!
2022-10-08T12:02:17Z
Hi@crumb,I was able to reproduce the problem and openedthis issueindiffusersto track and fix it.Thanks a lot for reaching out!
2022-10-08T12:05:56Z
I think this line might have something to do with it?github.comhuggingface/diffusers/blob/f3983d16eed57e46742d217363d8913bef7f748d/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py#L260uncond_input = self.tokenizer(uncond_tokens,padding="max_length",max_length=max_length,truncation=True,return_tensors="pt",)uncond_embeddings = self.text_encoder(uncond_input.input_ids.to(self.device))[0]# duplicate unconditional embeddings for each generation per promptuncond_embeddings = uncond_embeddings.repeat_interleave(batch_size * num_images_per_prompt, dim=0)# For classifier free guidance, we need to do two forward passes.# Here we concatenate the unconditional and text embeddings into a single batch# to avoid doing two forward passestext_embeddings = torch.cat([uncond_embeddings, text_embeddings])# get the initial random noise unless the user supplied it# Unlike in other pipelines, latents need to be generated in the target device# for 1-to-1 results reproducibility with the CompVis implementation.uncond embeddings will already be a list (the negative prompts) so repeating them again won’t help, then in the line after where it’s putting the negatives with the positives, the model is only seeing the negatives (for both the uncond and cond input). Could be wrong but with a quick glance that’s what I think’s happening.edit: oops you already said something like this in the issue, cheers!