"Diffusers Image Fill" guide

<p>

This guide was an idea I had for a while but was asked by <a href="https://github.com/pietrobolcato" rel="nofollow">pietrobolcato</a> <a href="https://github.com/huggingface/diffusers/discussions/7482#discussioncomment-10529470" rel="nofollow">here</a> so finally made the decision to do it before it gets too old or I forget it.</p>
<p>So the basic idea is to do a simple object remover or fill a selected part of the image that you want to change, for this we will use a controlnet and some easy techniques.</p>
<p>To be able to do this, we need to use two key models, one is the <a href="https://github.com/xinsir6/ControlNetPlus" rel="nofollow">ControlNetPlus Promax</a> and the second one is to use the lighting models, in this case, since I want to do photorealism, I'll use <a href="https://huggingface.co/SG161222/RealVisXL_V5.0_Lightning">RealVis 5.0 Lighting</a>.</p>
<p>The controlnet is not part of the diffusers core, but the official repository has all the instructions to make it work, you'll need the <code>StableDiffusionXLControlNetUnionPipeline</code>.</p>
<p>I also set up a space as a PoC of this guide, for this, I did a custom pipeline with just what we need to make it work. You can test it <a href="https://huggingface.co/spaces/OzzyGT/diffusers-image-fill">here</a>, if you use the app locally you can see the cool effect on how the image generates and fills the mask.</p>
<p align="center">
<img src="https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/diffusers_fill/2024-09-13_07-40.png" width="600"/>
</p>
<p>First we need an image, I downloaded some from <a href="https://unsplash.com/" rel="nofollow">unsplash.com</a>. Lets use a demo the car in the mountains, the original image is <a href="https://unsplash.com/photos/a-car-parked-on-a-dirt-road-near-a-mountain-OCQjiB4tG5c" rel="nofollow">here</a> which was taken by <a href="https://unsplash.com/@jeffersonsees" rel="nofollow">jeffersonsees</a></p>
<p>Since this guide uses a custom pipeline with a custom controlnet that is not part of the core, I can't post the full code or it will be to big, so I'll try to give the key parts of what's needed to make it work. Also I will simplify the process by assuming square images of 1024x1024 which is not ideal in a real world scenario, this should be adapted to be used with any image in any aspect ratio and resolution.</p>
<p>I'll use pillow to avoid doing too many conversions between formats, so let's make the image a square:</p>
<pre><code class="language-python"><span class="hljs-keyword">from</span> PIL <span class="hljs-keyword">import</span> Image

<span class="hljs-keyword">from</span> diffusers.utils <span class="hljs-keyword">import</span> load_image


source_image = load_image(
    <span class="hljs-string">"https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/diffusers_fill/jefferson-sees-OCQjiB4tG5c-unsplash.jpg"</span>
)

width, height = source_image.size
min_dimension = <span class="hljs-built_in">min</span>(width, height)

left = (width - min_dimension) / <span class="hljs-number">2</span>
top = (height - min_dimension) / <span class="hljs-number">2</span>
right = (width + min_dimension) / <span class="hljs-number">2</span>
bottom = (height + min_dimension) / <span class="hljs-number">2</span>

final_source = source_image.crop((left, top, right, bottom))
final_source = final_source.resize((<span class="hljs-number">1024</span>, <span class="hljs-number">1024</span>), Image.LANCZOS)
</code></pre>
<p>Then we need a mask, you can use any method to get it, you can use SAM2, BiRefNet or any of the newer models that lets you mask objects or it can be done manually, since this isn't about masking, I'll use the <a href="https://huggingface.co/spaces/stevhliu/inpaint-mask-maker">inpaint mask maker</a> to generate one.</p>
<p><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/OSahO1ONI-mwk-OJRc017.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/OSahO1ONI-mwk-OJRc017.png"/></a></p>
<p>Now that we have the two images, what we need to do is to delete the masked part from the original, the result is the image we're going to feed the controlnet.</p>
<pre><code class="language-python">mask = load_image(
    <span class="hljs-string">"https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/diffusers_fill/car_mask.png"</span>
).convert(<span class="hljs-string">"L"</span>)

inverted_mask = ImageChops.invert(mask)
cnet_image = final_source.copy()
cnet_image.putalpha(inverted_mask)
</code></pre>
<p>This is the first part of the technique, the second one is to use a lighting model with less steps than a non-distilled model and also to use the controlnet tile mode at full strength for the whole steps so it preserves as much of the original image as possible.</p>
<p>I'll assume for this part the following:</p>
<ul>
<li>You downloaded the ControlNetModel_Union model python file and have it in the same directory as the script.</li>
<li>You have downloaded the controlnet model weights locally and renamed the files accordingly.</li>
</ul>
<p>The reason for the second one is that the official repo doesn't have easy to use format for the promax version of the model, if you want to see how to load it directly from the hub, you can read the official repository or look at the app code in the space.</p>
<pre><code class="language-python"><span class="hljs-keyword">import</span> torch
<span class="hljs-keyword">from</span> controlnet_union <span class="hljs-keyword">import</span> ControlNetModel_Union
<span class="hljs-keyword">from</span> diffusers <span class="hljs-keyword">import</span> AutoencoderKL, StableDiffusionXLControlNetPipeline, TCDScheduler

vae = AutoencoderKL.from_pretrained(<span class="hljs-string">"madebyollin/sdxl-vae-fp16-fix"</span>, torch_dtype=torch.float16).to(<span class="hljs-string">"cuda"</span>)

controlnet_model = ControlNetModel_Union.from_pretrained(
    <span class="hljs-string">"./controlnet-union-sdxl-1.0"</span>,
    torch_dtype=torch.float16,
)

pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
    <span class="hljs-string">"SG161222/RealVisXL_V5.0_Lightning"</span>,
    torch_dtype=torch.float16,
    vae=vae,
    custom_pipeline=<span class="hljs-string">"OzzyGT/pipeline_sdxl_fill"</span>,
    controlnet=controlnet_model,
    variant=<span class="hljs-string">"fp16"</span>,
).to(<span class="hljs-string">"cuda"</span>)
pipe.scheduler = TCDScheduler.from_config(pipe.scheduler.config)

prompt = <span class="hljs-string">"high quality"</span>
(
    prompt_embeds,
    negative_prompt_embeds,
    pooled_prompt_embeds,
    negative_pooled_prompt_embeds,
) = pipe.encode_prompt(prompt, <span class="hljs-string">"cuda"</span>, <span class="hljs-literal">True</span>)

image = pipe(
    prompt_embeds=prompt_embeds,
    negative_prompt_embeds=negative_prompt_embeds,
    pooled_prompt_embeds=pooled_prompt_embeds,
    negative_pooled_prompt_embeds=negative_pooled_prompt_embeds,
    image=cnet_image,
)
</code></pre>
<p>With this, we get something like this image:</p>
<p align="center">
<img src="https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/diffusers_fill/weird_car_generation.png" width="600"/>
</p>
<p>I did on purpose a bad mask which leaves some details of the original car and makes the generation weird or bad, sometimes we get the borders of the car, or something like this one, I even got a buffalo!!!</p>
<p>So now that we know that the mask affects a lot the result, I'll do a more detailed one that I know it works with GIMP, since the mask will not be a pure white and black mask, we need to convert it to a binary mask.</p>
<pre><code class="language-python">mask = load_image(
    <span class="hljs-string">"https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/diffusers_fill/car_mask_good.png"</span>
).convert(<span class="hljs-string">"L"</span>)

binary_mask = mask.point(<span class="hljs-keyword">lambda</span> p: <span class="hljs-number">255</span> <span class="hljs-keyword">if</span> p &gt; <span class="hljs-number">0</span> <span class="hljs-keyword">else</span> <span class="hljs-number">0</span>)
inverted_mask = ImageChops.invert(binary_mask)
</code></pre>
<p>My custom pipeline does a lot of the stuff you normally have to do, under the hood, so this means, set the steps, scales and the appropiate mode and image for the controlnet. You can use it if you want, but keep in mind that it's very restrictive and can be used mostly for what I use it in this guide.</p>
<p>Also take note that I use the TCD Scheduler for this, since is what I think works best with the lighting models, I also tried using PAG but it made the results worse for some reason.</p>
<p>Now we get something like this:</p>
<p align="center">
<img src="https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/diffusers_fill/car_fill_generation.png" width="600"/>
</p>
<p>The last step for this guide is that, if you look closely, the image still gets changes, if the original image like this one has a good quality, you can see how it loses quality and some of the smaller details gets blurry, so to fix this we simply paste over the original alpha image and the beauty of this technique is that it merges seamesly, most people won't know it was inpainted if you don't tell them (I tested this).</p>
<div class="max-w-full overflow-auto">
<table>
<thead><tr>
<th>generation</th>
<th>merged</th>
</tr>
</thead><tbody><tr>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/GSQaYRmMGuXgfQBH2PxjM.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/GSQaYRmMGuXgfQBH2PxjM.png"/></a></td>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/aalKI7dkdFy8678zMVmUr.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/aalKI7dkdFy8678zMVmUr.png"/></a></td>
</tr>
</tbody>
</table>
</div>
<p>For this example, since the bushes has a lot of details, if you look closely you can see the transition, so dependending on you use case, it would be better to not do the final paste but again, most people won't even notice this.</p>
<p>Here are some more examples (you can use the space to test them too):</p>
<div class="max-w-full overflow-auto">
<table>
<thead><tr>
<th>original</th>
<th>fill</th>
</tr>
</thead><tbody><tr>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/JkKqqnDnDipaf6fbJOkOt.jpeg" rel="nofollow"><img alt="image/jpeg" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/JkKqqnDnDipaf6fbJOkOt.jpeg"/></a></td>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/Mh4QK-mzahK4Q1MiqF6DD.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/Mh4QK-mzahK4Q1MiqF6DD.png"/></a></td>
</tr>
<tr>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/dKuiQi5jKFTIpmn9Qu77T.jpeg" rel="nofollow"><img alt="image/jpeg" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/dKuiQi5jKFTIpmn9Qu77T.jpeg"/></a></td>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/tjZwj6gPljBD2Xq3ocFYC.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/tjZwj6gPljBD2Xq3ocFYC.png"/></a></td>
</tr>
</tbody>
</table>
</div>
<p>Credits for the images:</p>
<p>First one: <a href="https://unsplash.com/photos/a-blurry-photo-of-a-man-walking-past-a-restaurant-JuBjdYyxaAg" rel="nofollow">original</a> by <a href="https://unsplash.com/@leonardodbi" rel="nofollow">
Leonardo Iribe</a></p>
<p>Second one: <a href="https://unsplash.com/photos/a-woman-walking-across-a-bridge-over-a-body-of-water-BPoQtQl_6iE" rel="nofollow">original</a> by <a href="https://unsplash.com/@raymondpetrik" rel="nofollow">Raymond Petrik</a></p>
<p>There's also an added benefit that I plan to use for a new outpanting guide, and that is that you can expand an image, so this is ideal for generation a temporal background that we can use to add detail later.</p>
<div class="max-w-full overflow-auto">
<table>
<thead><tr>
<th>original</th>
<th>expanded</th>
</tr>
</thead><tbody><tr>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/SHJyLWcbEkl6qJAVyVYTX.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/SHJyLWcbEkl6qJAVyVYTX.png"/></a></td>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/WFiUGR4M-TZGxZqbEur1g.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/WFiUGR4M-TZGxZqbEur1g.png"/></a></td>
</tr>
</tbody>
</table>
</div>
<p>To improve the results, I encourage you to use some more advanced techniques like:</p>
<ul>
<li>Use differential diffusion to merge the seams with the original image</li>
<li>Upscale the masked final generation, use it with img2img to add more details and then paste it back on the original.</li>
<li>Adapt this with better models (SD3 or Flux) when the controlnets gets as good as this one.</li>
</ul>
<p>That's it for this guide, I hope it helps to learn how to use this awesome controlnet and to give you a headstart on how to get good quality images that you can use in your work.</p>
<p>The final full code with the final merge:</p>
<pre><code class="language-python"><span class="hljs-keyword">import</span> torch
<span class="hljs-keyword">from</span> PIL <span class="hljs-keyword">import</span> Image, ImageChops

<span class="hljs-keyword">from</span> controlnet_union <span class="hljs-keyword">import</span> ControlNetModel_Union
<span class="hljs-keyword">from</span> diffusers <span class="hljs-keyword">import</span> AutoencoderKL, StableDiffusionXLControlNetPipeline, TCDScheduler
<span class="hljs-keyword">from</span> diffusers.utils <span class="hljs-keyword">import</span> load_image


source_image = load_image(
    <span class="hljs-string">"https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/diffusers_fill/jefferson-sees-OCQjiB4tG5c-unsplash.jpg"</span>
)

width, height = source_image.size
min_dimension = <span class="hljs-built_in">min</span>(width, height)

left = (width - min_dimension) / <span class="hljs-number">2</span>
top = (height - min_dimension) / <span class="hljs-number">2</span>
right = (width + min_dimension) / <span class="hljs-number">2</span>
bottom = (height + min_dimension) / <span class="hljs-number">2</span>

final_source = source_image.crop((left, top, right, bottom))
final_source = final_source.resize((<span class="hljs-number">1024</span>, <span class="hljs-number">1024</span>), Image.LANCZOS).convert(<span class="hljs-string">"RGBA"</span>)

mask = load_image(
    <span class="hljs-string">"https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/diffusers_fill/car_mask_good.png"</span>
).convert(<span class="hljs-string">"L"</span>)

binary_mask = mask.point(<span class="hljs-keyword">lambda</span> p: <span class="hljs-number">255</span> <span class="hljs-keyword">if</span> p &gt; <span class="hljs-number">0</span> <span class="hljs-keyword">else</span> <span class="hljs-number">0</span>)
inverted_mask = ImageChops.invert(binary_mask)

alpha_image = Image.new(<span class="hljs-string">"RGBA"</span>, final_source.size, (<span class="hljs-number">0</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>))
cnet_image = Image.composite(final_source, alpha_image, inverted_mask)

vae = AutoencoderKL.from_pretrained(<span class="hljs-string">"madebyollin/sdxl-vae-fp16-fix"</span>, torch_dtype=torch.float16).to(<span class="hljs-string">"cuda"</span>)

controlnet_model = ControlNetModel_Union.from_pretrained(
    <span class="hljs-string">"./controlnet-union-sdxl-1.0"</span>,
    torch_dtype=torch.float16,
)

pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
    <span class="hljs-string">"SG161222/RealVisXL_V5.0_Lightning"</span>,
    torch_dtype=torch.float16,
    vae=vae,
    custom_pipeline=<span class="hljs-string">"OzzyGT/pipeline_sdxl_fill"</span>,
    controlnet=controlnet_model,
    variant=<span class="hljs-string">"fp16"</span>,
).to(<span class="hljs-string">"cuda"</span>)
pipe.scheduler = TCDScheduler.from_config(pipe.scheduler.config)

prompt = <span class="hljs-string">"high quality"</span>
(
    prompt_embeds,
    negative_prompt_embeds,
    pooled_prompt_embeds,
    negative_pooled_prompt_embeds,
) = pipe.encode_prompt(prompt, <span class="hljs-string">"cuda"</span>, <span class="hljs-literal">True</span>)

image = pipe(
    prompt_embeds=prompt_embeds,
    negative_prompt_embeds=negative_prompt_embeds,
    pooled_prompt_embeds=pooled_prompt_embeds,
    negative_pooled_prompt_embeds=negative_pooled_prompt_embeds,
    image=cnet_image,
)

image = image.convert(<span class="hljs-string">"RGBA"</span>)
cnet_image.paste(image, (<span class="hljs-number">0</span>, <span class="hljs-number">0</span>), binary_mask)

cnet_image.save(<span class="hljs-string">"final_generation.png"</span>)
</code></pre>
