Train_dreambooth_lora is not working!
Describe the bugHello!I tried to use train_dreambooth_lora but after finishing training, I can’t see my face on the result image that I trained!After making an image, that is a very random instance like a Car or train or another human face that is not me.what is the problem here?!ReproductionThis is my training parameters:MODEL_NAME="XpucT/Deliberate"
INSTANCE_DIR="datasets/instance-images/1fdfdf359622486c94e742e3e1a975cb/"
OUTPUT_DIR="output-models/1fdfdf359622486c94e742e3e1a975cb/"
CLASS_DIR="datasets/class-man/"

!accelerate launch train_dreambooth_lora.py \
  --pretrained_model_name_or_path=$MODEL_NAME  \
  --instance_data_dir=$INSTANCE_DIR \
  --class_data_dir=$CLASS_DIR \
  --output_dir=$OUTPUT_DIR \
  --instance_prompt="<hosein>" \
  --class_prompt="photo of a man" \
  --resolution=512 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=1 \
  --checkpointing_steps=100 \
  --with_prior_preservation \
  --prior_loss_weight=0.1 \
  --learning_rate=1e-7 \
  --lr_scheduler="constant" \
  --lr_warmup_steps=80 \
  --num_class_images=120 \
  --max_train_steps=1000 \
  --seed="9274" \
  --mixed_precision="fp16" \
  --train_text_encoder \
  --use_8bit_adam \
  --enable_xformers_memory_efficient_attentionNow make a pipeline and apply Lora weights:import torch
import random
from torch import autocast
from diffusers import StableDiffusionPipeline, DDIMScheduler, DPMSolverMultistepScheduler
from IPython.display import display

model_path = "output-models/1fdfdf359622486c94e742e3e1a975cb/"


def get_base_model(model_base):
    pipe = StableDiffusionPipeline.from_pretrained(model_base, torch_dtype=torch.float16, safety_checker=None)
    pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
    pipe.to("cuda")
    return pipe

def assign_lora_weight(pipe, lora_weight_id):
    pipe.unet.load_attn_procs(lora_weight_id)
    return pipe

# get base model for one time
base_pipe = get_base_model("XpucT/Deliberate")

# lora-layer-painter-wise.
hosein_pip = assign_lora_weight(base_pipe, lora_weight_id=model_path)Now I want make new image from my face:image = hosein_pip(    
    "<hosein>",
    height=512,
    width=512,
    negative_prompt="blender, cropped, lowres, poorly drawn face, out of frame, poorly drawn hands, blurry, bad art, blurred, text, watermark, disfigured, blurry background, deformed, closed eyes, animation, CGI ",
    num_inference_steps=25,
    guidance_scale=7.5,
    cross_attention_kwargs={"scale": 2.5},
).images[0]

image.save(f"result/{random.randint(1,1000000)}.jpg")LogsNo responseSystem Infogithub.comdiffusers/examples/dreambooth at main · ShivamShrirao/diffusersmain/examples/dreambooth🤗 Diffusers: State-of-the-art diffusion models for image and audio generation in PyTorch
2023-05-02T20:45:03Z
hey@admin2684we had some issues with people using xformers 0.0.16. Are you per-chance using that version?