Outpainting II - Differential Diffusion

<p>

This is the third guide about outpainting, if you want to read about the other methods here they are:</p>
<ul>
<li><a href="https://huggingface.co/blog/OzzyGT/outpainting-controlnet">Outpainting I - Controlnet version</a></li>
<li><a href="https://huggingface.co/blog/OzzyGT/outpainting-inpaint-model">Outpainting III - Inpaint Model</a></li>
</ul>
<p>In this guide I'll explore how to do outpainting with <a href="https://github.com/exx8/differential-diffusion" rel="nofollow">differential diffusion</a> in depth going though each of the steps I did to get good results.</p>
<p>I’ll start with a non-square image that has a depth of field (bokeh) to make it more difficult. When they have this kind of background, it’s really easy to see the seams. This is an image that I grabbed from <a href="https://unsplash.com/photos/a-man-in-a-white-hoodie-holding-a-camera-Qj9kZ05paJI" rel="nofollow">Unsplash</a>:</p>
<p align="center">
<img src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/d2B1x4BAWf2kzJzHBBeFU.jpeg" width="400"/>
</p>
<p>So, the first task is to make it a square image (expand it) so we can keep making it bigger, but I’ll generate images of 1024x1024 each time as this is the optimal resolution for SDXL.</p>
<p>Then, I’ll test the result if I just use the new area with a gray background. But to also do that, we need to create a mask that can work with differential diffusion. For this, I’ll move the margin 50 pixels to the left and apply a blur filter. This helps to smooth the transition.</p>
<div class="max-w-full overflow-auto">
<table>
<thead><tr>
<th>squared image</th>
<th>mask</th>
<th>blurred mask</th>
</tr>
</thead><tbody><tr>
<td><img src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/cq6MwMDP0EULAcePp3_O3.png" width="240"/></td>
<td><img src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/6LZSgargIzkyCEec5Ids9.png" width="240"/></td>
<td><img src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/viDk7p8KfvJt59G_bfU-N.png" width="240"/></td>
</tr>
</tbody>
</table>
</div>
<p>We're going to use the community pipeline <code>StableDiffusionXLDifferentialImg2ImgPipeline</code> and it's loaded like this:</p>
<pre><code class="language-python">pipeline = StableDiffusionXLPipeline.from_pretrained(
    <span class="hljs-string">"stabilityai/stable-diffusion-xl-base-1.0"</span>,
    torch_dtype=torch.float16,
    variant=<span class="hljs-string">"fp16"</span>,
    custom_pipeline=<span class="hljs-string">"pipeline_stable_diffusion_xl_differential_img2img"</span>,
).to(<span class="hljs-string">"cuda"</span>)

image = pipeline(
    prompt=prompt=<span class="hljs-string">""</span>,
    negative_prompt=<span class="hljs-string">""</span>,
    width=<span class="hljs-number">1024</span>,
    height=<span class="hljs-number">1024</span>,
    guidance_scale=<span class="hljs-number">6.0</span>,
    num_inference_steps=<span class="hljs-number">25</span>,
    original_image=image,
    image=image,
    strength=<span class="hljs-number">1.0</span>,
    <span class="hljs-built_in">map</span>=mask,
).images[<span class="hljs-number">0</span>]
</code></pre>
<p>At this point, If we generate the image without a prompt, the model will think that the gray area is a gray object like a wall:</p>
<div class="max-w-full overflow-auto">
<table>
<thead><tr>
<th></th>
<th></th>
</tr>
</thead><tbody><tr>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/I3RqrhTHGsF_HymdvR7KE.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/I3RqrhTHGsF_HymdvR7KE.png"/></a></td>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/B56-oBfA0QFDfEyu11XRx.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/B56-oBfA0QFDfEyu11XRx.png"/></a></td>
</tr>
</tbody>
</table>
</div>
<p>If this is used by someone who knows how to draw, that person could make a rough drawing and generate the image. Since I’m not that person, I’ll need to think of a prompt for the new outpainting area.</p>
<p>For this, we can create it ourselves, use an online chatbot like GPT-4V or Bing Chat, or a local VLLM like Llava. Personally, I always like to use local VLLMs, and this one got my attention: <a href="https://huggingface.co/internlm/internlm-xcomposer2-vl-7b-4bit">internlm-xcomposer2-vl-7b-4bit</a> because it works really well, even with just the 4-bit version.</p>
<p>This is what I got:</p>
<p><code>The image captures a man standing on the shore of a body of water, possibly a lake or river. He is wearing a white hoodie with the word 'evolution' written across it and khaki pants. A green backpack is slung over his shoulders, and he holds a camera in his hands. The backdrop features a mountain range under a clear sky.</code></p>
<p>for a comparison, this is what bing gave me:</p>
<p><code>The image depicts a photographer, dressed in outdoor gear and holding a professional camera, set against a stunning backdrop of a serene lake and snow-capped mountains. It’s a beautiful blend of human activity and natural beauty.</code></p>
<p>When doing inpainting or outpainting, the prompt is really important, as an example, these are the results with both prompts:</p>
<div class="max-w-full overflow-auto">
<table>
<thead><tr>
<th>XComposer2</th>
<th>XComposer2</th>
</tr>
</thead><tbody><tr>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/UEv34tUgaZe2iK3whfJHE.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/UEv34tUgaZe2iK3whfJHE.png"/></a></td>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/zut9yVjpopcntfDak1Rfz.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/zut9yVjpopcntfDak1Rfz.png"/></a></td>
</tr>
<tr>
<td>Bing</td>
<td>Bing</td>
</tr>
<tr>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/Zfwqu2PvEV1Yl7HKNohA8.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/Zfwqu2PvEV1Yl7HKNohA8.png"/></a></td>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/JXZJe8PddDV4VjErR-LBt.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/JXZJe8PddDV4VjErR-LBt.png"/></a></td>
</tr>
</tbody>
</table>
</div>
<p>For this specific image, and perhaps for SDXL in general, the prompt generated by XComposer2 is better because it describes the image without exaggerated words like <code>stunning backdrop</code>, <code>beautiful blend</code> or <code>natural beauty</code>.</p>
<p>Taking the XComposer2 prompt and fixing the seed, let’s see how differential diffusion works.</p>
<div class="max-w-full overflow-auto">
<table>
<thead><tr>
<th>normal mask</th>
<th>blurred mask</th>
</tr>
</thead><tbody><tr>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/LrKK0JAOsW4Pj_Hnu0cde.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/LrKK0JAOsW4Pj_Hnu0cde.png"/></a></td>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/UEv34tUgaZe2iK3whfJHE.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/UEv34tUgaZe2iK3whfJHE.png"/></a></td>
</tr>
<tr>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/pW3WyohlhKW3sFP-iEurG.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/pW3WyohlhKW3sFP-iEurG.png"/></a></td>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/zut9yVjpopcntfDak1Rfz.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/zut9yVjpopcntfDak1Rfz.png"/></a></td>
</tr>
</tbody>
</table>
</div>
<p>We can see that differential diffusion blends the outpaint better with the original image, even when they’re totally different. Let’s see what happens when we increase the blur.</p>
<div class="max-w-full overflow-auto">
<table>
<thead><tr>
<th>blur radius 20</th>
<th>blur radius 40</th>
</tr>
</thead><tbody><tr>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/UEv34tUgaZe2iK3whfJHE.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/UEv34tUgaZe2iK3whfJHE.png"/></a></td>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/3UnIA9kEl48TzR1E9xTRf.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/3UnIA9kEl48TzR1E9xTRf.png"/></a></td>
</tr>
<tr>
<td>blur radius 80</td>
<td>blur radius 100</td>
</tr>
<tr>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/Zwp66u_PyVk2v43ox0N42.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/Zwp66u_PyVk2v43ox0N42.png"/></a></td>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/WuT5GikRpvv-Goi_BxOwM.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/WuT5GikRpvv-Goi_BxOwM.png"/></a></td>
</tr>
</tbody>
</table>
</div>
<p>Now, we can clearly see why differential diffusion is a really good method for inpainting and outpainting. With this outpaint area and with a blur of 80 or 100, the only reason we can see the seam is because of the color difference. Just take into account that the larger the blur and the area, the more the original image will change.</p>
<p>To solve this problem with the color, or at least attenuate it, we need to fill the new area with something else. Something that helps the model better understand what we want in the new area.</p>
<p>There are several techniques that can do this. Each of them helps but has different uses. For example, <a href="https://gfx.cs.princeton.edu/pubs/Barnes_2009_PAR/" rel="nofollow">PatchMatch</a> or <a href="https://github.com/advimman/lama" rel="nofollow">LaMa</a> helps with inpainting since they remove the content and fill it with a new one. For this use case, those don’t work that well because the area they need to fill is too big and completely new. So, I’ll use the <a href="https://docs.opencv.org/4.x/df/d3d/tutorial_py_inpainting.html" rel="nofollow">OpenCV</a> ones. In this case, I like the result with the <code>telea algorithm</code>.</p>
<p>To use this method, it’s necessary to install OpenCV for Python:</p>
<pre><code class="language-bash">pip install opencv-python
</code></pre>
<p>It’s not a good idea to convert images between multiple libraries because it can result in a loss of quality. So, for this, I’ll convert all the functions to <code>OpenCV</code>. The only major difference is the blur. To obtain an effect similar to <code>Pillow</code>, we need to use a much higher value. In this case, a blur radius of 500.</p>
<p>The mask we need for the Telea inpaint must be the same size as the original mask, without the offset, since that’s the area we want to replace.</p>
<p align="center">
<img src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/FxSD3uQPv_clPjAIekxJ-.png" width="400"/>
</p>
<p>We need the model to use this information. Normally, with an inpainting model or with a normal image-to-image model, we decrease the value of <code>strength</code>. But with differential, we can keep this value at the maximum and just make the mask lighter. I’ll use a dark gray for this.</p>
<div class="max-w-full overflow-auto">
<table>
<thead><tr>
<th></th>
<th></th>
</tr>
</thead><tbody><tr>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/KDd9FOHvAsUqr1B4VqizZ.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/KDd9FOHvAsUqr1B4VqizZ.png"/></a></td>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/mj58J-RkzarhqCFg5LQvP.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/mj58J-RkzarhqCFg5LQvP.png"/></a></td>
</tr>
<tr>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/QKwhkSK3frEoQbwmJYU9D.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/QKwhkSK3frEoQbwmJYU9D.png"/></a></td>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/PnMII7_IhhAvXjqpaDsni.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/PnMII7_IhhAvXjqpaDsni.png"/></a></td>
</tr>
</tbody>
</table>
</div>
<p>Now, we have some good results, but I still see two problems. We can still see the seam because there’s a slight difference in the colors, and we depend on the prompt to do this. If we pass the wrong prompt (which is highly probable if you use a VLLM), the outpainting will be bad.</p>
<p>To fix both of these problems, we’re going to use IP Adapters. This is pretty obvious; there’s no better way to tell the model the details of the original image than an Image Prompt.</p>
<p>The only problem we have right now is that the original image is not a square image, and IP Adapters only work with square images. There’s a solution to this proposed by the original authors that involves resizing and padding the image, but that would make it feed that information to the model, and we don’t want that because we’re precisely trying to paint that area.</p>
<p>Since we don’t really need to give it an exact composition and we can feed multiple images to the IP Adapter, what we’re going to do is to slice the original image into squares and feed those to the IP Adapter. For this, it’s better to use the larger initial image and then resize each square down to 224x224, which is the size they need.</p>
<p align="center">
<img src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/5a8nHJvw93I32zKPMyn2U.jpeg" width="400"/>
</p>
<p>This function can do this:</p>
<pre><code class="language-python"><span class="hljs-keyword">def</span> <span class="hljs-title function_">slice_image</span>(<span class="hljs-params">image</span>):
    height, width, _ = image.shape
    slice_size = <span class="hljs-built_in">min</span>(width // <span class="hljs-number">2</span>, height // <span class="hljs-number">3</span>)

    slices = []

    <span class="hljs-keyword">for</span> h <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(<span class="hljs-number">3</span>):
        <span class="hljs-keyword">for</span> w <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(<span class="hljs-number">2</span>):
            left = w * slice_size
            upper = h * slice_size
            right = left + slice_size
            lower = upper + slice_size

            <span class="hljs-keyword">if</span> w == <span class="hljs-number">1</span> <span class="hljs-keyword">and</span> right &gt; width:
                left -= right - width
                right = width
            <span class="hljs-keyword">if</span> h == <span class="hljs-number">2</span> <span class="hljs-keyword">and</span> lower &gt; height:
                upper -= lower - height
                lower = height

            <span class="hljs-built_in">slice</span> = image[upper:lower, left:right]
            slices.append(<span class="hljs-built_in">slice</span>)

    <span class="hljs-keyword">return</span> slices
</code></pre>
<p>These are the sliced images we get with it:</p>
<div class="max-w-full overflow-auto">
<table>
<thead><tr>
<th></th>
<th></th>
<th></th>
</tr>
</thead><tbody><tr>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/tWlAKHoUZTf5ZqhLWORJa.jpeg" rel="nofollow"><img alt="image/jpeg" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/tWlAKHoUZTf5ZqhLWORJa.jpeg"/></a></td>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/3PJUSNlnAwV8N47XEqKaV.jpeg" rel="nofollow"><img alt="image/jpeg" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/3PJUSNlnAwV8N47XEqKaV.jpeg"/></a></td>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/6MiUnq0bsQ52NT64Vg3Ue.jpeg" rel="nofollow"><img alt="image/jpeg" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/6MiUnq0bsQ52NT64Vg3Ue.jpeg"/></a></td>
</tr>
<tr>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/1do0Tbj2jJHf849L96DdC.jpeg" rel="nofollow"><img alt="image/jpeg" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/1do0Tbj2jJHf849L96DdC.jpeg"/></a></td>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/QdPKacnL9VmfESx2UjyP0.jpeg" rel="nofollow"><img alt="image/jpeg" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/QdPKacnL9VmfESx2UjyP0.jpeg"/></a></td>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/6XackYEJpxiV-s_h1pmax.jpeg" rel="nofollow"><img alt="image/jpeg" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/6XackYEJpxiV-s_h1pmax.jpeg"/></a></td>
</tr>
</tbody>
</table>
</div>
<p>Without a prompt and since we're feeding these images to the IP Adapter, we can lower the CFG to about 4.0</p>
<div class="max-w-full overflow-auto">
<table>
<thead><tr>
<th></th>
<th></th>
</tr>
</thead><tbody><tr>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/a9HwtuSXiJk9T2Io62Yjw.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/a9HwtuSXiJk9T2Io62Yjw.png"/></a></td>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/1J0C_sxlD5PDALS0UWjyv.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/1J0C_sxlD5PDALS0UWjyv.png"/></a></td>
</tr>
<tr>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/nnilpwvde14mzF4t7HoRp.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/nnilpwvde14mzF4t7HoRp.png"/></a></td>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/mEJTo5O4o7F_LbWzzcrMb.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/mEJTo5O4o7F_LbWzzcrMb.png"/></a></td>
</tr>
</tbody>
</table>
</div>
<p>Sometimes we can get images that still have seams but most of the time they're good and we fixed the color difference because the IP Adapter gave that information to the model.</p>
<p>Now we have a script that can expand portrait/landscape images without the need of a prompt, these are tests I did with other images:</p>
<div class="max-w-full overflow-auto">
<table>
<thead><tr>
<th>original</th>
<th>expanded</th>
</tr>
</thead><tbody><tr>
<td><img src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/LWUfqp84eQBbQuoq_4rns.jpeg" width="300"/></td>
<td><img src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/8YCJz0SZjkNpzYOJEpw_K.png" width="400"/></td>
</tr>
<tr>
<td><img src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/fLfW9zMPShz4MsKTBXDF0.jpeg" width="300"/></td>
<td><img src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/iaEFuj6o7Gh1L-PBFheHq.png" width="400"/></td>
</tr>
<tr>
<td><img src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/juaTntZtcDhjeIs6tYFWz.jpeg" width="300"/></td>
<td><img src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/MztS3W_TqCiodj4QyLfOj.png" width="400"/></td>
</tr>
<tr>
<td><img src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/hha3zjPLQzdFsuWkn4cY0.png" width="300"/></td>
<td><img src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/ui-YG_y_J7F7VEgDQ7LVd.png" width="400"/></td>
</tr>
</tbody>
</table>
</div>
<p>With this method, if the subject that you want to preserve is positioned at the border, it will change a little because we’re using a blurred mask. If you don’t want this, you can try to reduce the blur and the offset of the mask. If that doesn’t work, the only alternative is to use an inpainting model.</p>
<p>There are also some images that won’t work with this method. For example, this one:</p>
<div class="max-w-full overflow-auto">
<table>
<thead><tr>
<th>original</th>
<th>expanded</th>
</tr>
</thead><tbody><tr>
<td><img src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/gf_Mt805NoRtNtK9_Pi-M.jpeg" width="300"/></td>
<td><img src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/EO-kpsOxWlbbBPFfn6YSQ.png" width="400"/></td>
</tr>
</tbody>
</table>
</div>
<p>That’s because we only have half of the subject, and also the Telea algorithm expands the colors to the right. In this case, we can give it a little help with the prompt. I’ll use "colored eggs inside a round nest on a table":</p>
<div class="max-w-full overflow-auto">
<table>
<thead><tr>
<th></th>
<th></th>
</tr>
</thead><tbody><tr>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/inCYg9cHMorqZmgT0X82Q.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/inCYg9cHMorqZmgT0X82Q.png"/></a></td>
<td><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/saa5-9EniGkOI9-TKskMe.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/saa5-9EniGkOI9-TKskMe.png"/></a></td>
</tr>
</tbody>
</table>
</div>
<p>The model that you use is also very important. Some models perform outpainting better, while others are better suited for realistic photos or for specific genres like anime, fantasy, etc.</p>
<p>Now, the only thing we have left to do is to create really large outpaints:</p>
<p><a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/Pe2eNLRXZw48TwOcMeiVG.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/Pe2eNLRXZw48TwOcMeiVG.png"/></a>
<a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/PM_05wPnU28OTOw50BZal.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/PM_05wPnU28OTOw50BZal.png"/></a>
<a href="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/9_j7hrBndwECXfbkbxQRN.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/63df091910678851bb0cd0e0/9_j7hrBndwECXfbkbxQRN.png"/></a></p>
<p>This is the complete code. First, I make the image a square and then expand it. You can choose the direction in which to expand it. Please note that this is just a code example. You’ll need to modify it to suit your needs, but hopefully, this will help you get started with this kind of outpainting using diffusers and differential diffusion.</p>
<pre><code class="language-python"><span class="hljs-keyword">import</span> random
<span class="hljs-keyword">import</span> urllib.request

<span class="hljs-keyword">import</span> cv2
<span class="hljs-keyword">import</span> numpy <span class="hljs-keyword">as</span> np
<span class="hljs-keyword">import</span> torch

<span class="hljs-keyword">from</span> diffusers <span class="hljs-keyword">import</span> DPMSolverMultistepScheduler, StableDiffusionXLPipeline


<span class="hljs-keyword">def</span> <span class="hljs-title function_">merge_images</span>(<span class="hljs-params">original, new_image, offset, direction</span>):
    <span class="hljs-keyword">if</span> direction <span class="hljs-keyword">in</span> [<span class="hljs-string">"left"</span>, <span class="hljs-string">"right"</span>]:
        merged_image = np.zeros((original.shape[<span class="hljs-number">0</span>], original.shape[<span class="hljs-number">1</span>] + offset, <span class="hljs-number">3</span>), dtype=np.uint8)
    <span class="hljs-keyword">elif</span> direction <span class="hljs-keyword">in</span> [<span class="hljs-string">"top"</span>, <span class="hljs-string">"bottom"</span>]:
        merged_image = np.zeros((original.shape[<span class="hljs-number">0</span>] + offset, original.shape[<span class="hljs-number">1</span>], <span class="hljs-number">3</span>), dtype=np.uint8)

    <span class="hljs-keyword">if</span> direction == <span class="hljs-string">"left"</span>:
        merged_image[:, offset:] = original
        merged_image[:, : new_image.shape[<span class="hljs-number">1</span>]] = new_image
    <span class="hljs-keyword">elif</span> direction == <span class="hljs-string">"right"</span>:
        merged_image[:, : original.shape[<span class="hljs-number">1</span>]] = original
        merged_image[:, original.shape[<span class="hljs-number">1</span>] + offset - new_image.shape[<span class="hljs-number">1</span>] : original.shape[<span class="hljs-number">1</span>] + offset] = new_image
    <span class="hljs-keyword">elif</span> direction == <span class="hljs-string">"top"</span>:
        merged_image[offset:, :] = original
        merged_image[: new_image.shape[<span class="hljs-number">0</span>], :] = new_image
    <span class="hljs-keyword">elif</span> direction == <span class="hljs-string">"bottom"</span>:
        merged_image[: original.shape[<span class="hljs-number">0</span>], :] = original
        merged_image[original.shape[<span class="hljs-number">0</span>] + offset - new_image.shape[<span class="hljs-number">0</span>] : original.shape[<span class="hljs-number">0</span>] + offset, :] = new_image

    <span class="hljs-keyword">return</span> merged_image


<span class="hljs-keyword">def</span> <span class="hljs-title function_">slice_image</span>(<span class="hljs-params">image</span>):
    height, width, _ = image.shape
    slice_size = <span class="hljs-built_in">min</span>(width // <span class="hljs-number">2</span>, height // <span class="hljs-number">3</span>)

    slices = []

    <span class="hljs-keyword">for</span> h <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(<span class="hljs-number">3</span>):
        <span class="hljs-keyword">for</span> w <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(<span class="hljs-number">2</span>):
            left = w * slice_size
            upper = h * slice_size
            right = left + slice_size
            lower = upper + slice_size

            <span class="hljs-keyword">if</span> w == <span class="hljs-number">1</span> <span class="hljs-keyword">and</span> right &gt; width:
                left -= right - width
                right = width
            <span class="hljs-keyword">if</span> h == <span class="hljs-number">2</span> <span class="hljs-keyword">and</span> lower &gt; height:
                upper -= lower - height
                lower = height

            <span class="hljs-built_in">slice</span> = image[upper:lower, left:right]
            slices.append(<span class="hljs-built_in">slice</span>)

    <span class="hljs-keyword">return</span> slices


<span class="hljs-keyword">def</span> <span class="hljs-title function_">process_image</span>(<span class="hljs-params"></span>
<span class="hljs-params">    image,</span>
<span class="hljs-params">    fill_color=(<span class="hljs-params"><span class="hljs-number">0</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span></span>),</span>
<span class="hljs-params">    mask_offset=<span class="hljs-number">50</span>,</span>
<span class="hljs-params">    blur_radius=<span class="hljs-number">500</span>,</span>
<span class="hljs-params">    expand_pixels=<span class="hljs-number">256</span>,</span>
<span class="hljs-params">    direction=<span class="hljs-string">"left"</span>,</span>
<span class="hljs-params">    inpaint_mask_color=<span class="hljs-number">50</span>,</span>
<span class="hljs-params">    max_size=<span class="hljs-number">1024</span>,</span>
<span class="hljs-params"></span>):
    height, width = image.shape[:<span class="hljs-number">2</span>]

    new_height = height + (expand_pixels <span class="hljs-keyword">if</span> direction <span class="hljs-keyword">in</span> [<span class="hljs-string">"top"</span>, <span class="hljs-string">"bottom"</span>] <span class="hljs-keyword">else</span> <span class="hljs-number">0</span>)
    new_width = width + (expand_pixels <span class="hljs-keyword">if</span> direction <span class="hljs-keyword">in</span> [<span class="hljs-string">"left"</span>, <span class="hljs-string">"right"</span>] <span class="hljs-keyword">else</span> <span class="hljs-number">0</span>)

    <span class="hljs-keyword">if</span> new_height &gt; max_size:
        <span class="hljs-comment"># If so, crop the image from the opposite side</span>
        <span class="hljs-keyword">if</span> direction == <span class="hljs-string">"top"</span>:
            image = image[:max_size, :]
        <span class="hljs-keyword">elif</span> direction == <span class="hljs-string">"bottom"</span>:
            image = image[new_height - max_size :, :]
        new_height = max_size

    <span class="hljs-keyword">if</span> new_width &gt; max_size:
        <span class="hljs-comment"># If so, crop the image from the opposite side</span>
        <span class="hljs-keyword">if</span> direction == <span class="hljs-string">"left"</span>:
            image = image[:, :max_size]
        <span class="hljs-keyword">elif</span> direction == <span class="hljs-string">"right"</span>:
            image = image[:, new_width - max_size :]
        new_width = max_size

    height, width = image.shape[:<span class="hljs-number">2</span>]

    new_image = np.full((new_height, new_width, <span class="hljs-number">3</span>), fill_color, dtype=np.uint8)
    mask = np.full_like(new_image, <span class="hljs-number">255</span>, dtype=np.uint8)
    inpaint_mask = np.full_like(new_image, <span class="hljs-number">0</span>, dtype=np.uint8)

    mask = cv2.cvtColor(mask, cv2.COLOR_BGR2GRAY)
    inpaint_mask = cv2.cvtColor(inpaint_mask, cv2.COLOR_BGR2GRAY)

    <span class="hljs-keyword">if</span> direction == <span class="hljs-string">"left"</span>:
        new_image[:, expand_pixels:] = image[:, : max_size - expand_pixels]
        mask[:, : expand_pixels + mask_offset] = inpaint_mask_color
        inpaint_mask[:, :expand_pixels] = <span class="hljs-number">255</span>
    <span class="hljs-keyword">elif</span> direction == <span class="hljs-string">"right"</span>:
        new_image[:, :width] = image
        mask[:, width - mask_offset :] = inpaint_mask_color
        inpaint_mask[:, width:] = <span class="hljs-number">255</span>
    <span class="hljs-keyword">elif</span> direction == <span class="hljs-string">"top"</span>:
        new_image[expand_pixels:, :] = image[: max_size - expand_pixels, :]
        mask[: expand_pixels + mask_offset, :] = inpaint_mask_color
        inpaint_mask[:expand_pixels, :] = <span class="hljs-number">255</span>
    <span class="hljs-keyword">elif</span> direction == <span class="hljs-string">"bottom"</span>:
        new_image[:height, :] = image
        mask[height - mask_offset :, :] = inpaint_mask_color
        inpaint_mask[height:, :] = <span class="hljs-number">255</span>

    <span class="hljs-comment"># mask blur</span>
    <span class="hljs-keyword">if</span> blur_radius % <span class="hljs-number">2</span> == <span class="hljs-number">0</span>:
        blur_radius += <span class="hljs-number">1</span>
    mask = cv2.GaussianBlur(mask, (blur_radius, blur_radius), <span class="hljs-number">0</span>)

    <span class="hljs-comment"># telea inpaint</span>
    _, mask_np = cv2.threshold(inpaint_mask, <span class="hljs-number">128</span>, <span class="hljs-number">255</span>, cv2.THRESH_BINARY | cv2.THRESH_OTSU)
    inpaint = cv2.inpaint(new_image, mask_np, <span class="hljs-number">3</span>, cv2.INPAINT_TELEA)

    <span class="hljs-comment"># convert image to tensor</span>
    inpaint = cv2.cvtColor(inpaint, cv2.COLOR_BGR2RGB)
    inpaint = torch.from_numpy(inpaint).permute(<span class="hljs-number">2</span>, <span class="hljs-number">0</span>, <span class="hljs-number">1</span>).<span class="hljs-built_in">float</span>()
    inpaint = inpaint / <span class="hljs-number">127.5</span> - <span class="hljs-number">1</span>
    inpaint = inpaint.unsqueeze(<span class="hljs-number">0</span>).to(<span class="hljs-string">"cuda"</span>)

    <span class="hljs-comment"># convert mask to tensor</span>
    mask = torch.from_numpy(mask)
    mask = mask.unsqueeze(<span class="hljs-number">0</span>).<span class="hljs-built_in">float</span>() / <span class="hljs-number">255.0</span>
    mask = mask.to(<span class="hljs-string">"cuda"</span>)

    <span class="hljs-keyword">return</span> inpaint, mask


<span class="hljs-keyword">def</span> <span class="hljs-title function_">image_resize</span>(<span class="hljs-params">image, new_size=<span class="hljs-number">1024</span></span>):
    height, width = image.shape[:<span class="hljs-number">2</span>]

    aspect_ratio = width / height
    new_width = new_size
    new_height = new_size

    <span class="hljs-keyword">if</span> aspect_ratio != <span class="hljs-number">1</span>:
        <span class="hljs-keyword">if</span> width &gt; height:
            new_height = <span class="hljs-built_in">int</span>(new_size / aspect_ratio)
        <span class="hljs-keyword">else</span>:
            new_width = <span class="hljs-built_in">int</span>(new_size * aspect_ratio)

    image = cv2.resize(image, (new_width, new_height), interpolation=cv2.INTER_LANCZOS4)

    <span class="hljs-keyword">return</span> image


pipeline = StableDiffusionXLPipeline.from_pretrained(
    <span class="hljs-string">"SG161222/RealVisXL_V4.0"</span>,
    torch_dtype=torch.float16,
    variant=<span class="hljs-string">"fp16"</span>,
    custom_pipeline=<span class="hljs-string">"pipeline_stable_diffusion_xl_differential_img2img"</span>,
).to(<span class="hljs-string">"cuda"</span>)
pipeline.scheduler = DPMSolverMultistepScheduler.from_config(pipeline.scheduler.config, use_karras_sigmas=<span class="hljs-literal">True</span>)

pipeline.load_ip_adapter(
    <span class="hljs-string">"h94/IP-Adapter"</span>,
    subfolder=<span class="hljs-string">"sdxl_models"</span>,
    weight_name=[
        <span class="hljs-string">"ip-adapter-plus_sdxl_vit-h.safetensors"</span>,
    ],
    image_encoder_folder=<span class="hljs-string">"models/image_encoder"</span>,
)
pipeline.set_ip_adapter_scale(<span class="hljs-number">0.1</span>)


<span class="hljs-keyword">def</span> <span class="hljs-title function_">generate_image</span>(<span class="hljs-params">prompt, negative_prompt, image, mask, ip_adapter_image, seed: <span class="hljs-built_in">int</span> = <span class="hljs-literal">None</span></span>):
    <span class="hljs-keyword">if</span> seed <span class="hljs-keyword">is</span> <span class="hljs-literal">None</span>:
        seed = random.randint(<span class="hljs-number">0</span>, <span class="hljs-number">2</span>**<span class="hljs-number">32</span> - <span class="hljs-number">1</span>)

    generator = torch.Generator(device=<span class="hljs-string">"cpu"</span>).manual_seed(seed)

    image = pipeline(
        prompt=prompt,
        negative_prompt=negative_prompt,
        width=<span class="hljs-number">1024</span>,
        height=<span class="hljs-number">1024</span>,
        guidance_scale=<span class="hljs-number">4.0</span>,
        num_inference_steps=<span class="hljs-number">25</span>,
        original_image=image,
        image=image,
        strength=<span class="hljs-number">1.0</span>,
        <span class="hljs-built_in">map</span>=mask,
        generator=generator,
        ip_adapter_image=[ip_adapter_image],
        output_type=<span class="hljs-string">"np"</span>,
    ).images[<span class="hljs-number">0</span>]

    image = (image * <span class="hljs-number">255</span>).astype(np.uint8)
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

    <span class="hljs-keyword">return</span> image


prompt = <span class="hljs-string">""</span>
negative_prompt = <span class="hljs-string">""</span>
direction = <span class="hljs-string">"right"</span>  <span class="hljs-comment"># left, right, top, bottom</span>
inpaint_mask_color = <span class="hljs-number">50</span>  <span class="hljs-comment"># lighter use more of the Telea inpainting</span>
expand_pixels = <span class="hljs-number">256</span>  <span class="hljs-comment"># I recommend to don't go more than half of the picture so it has context</span>
times_to_expand = <span class="hljs-number">4</span>

url = <span class="hljs-string">"https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/differential/photo-1711580377289-eecd23d00370.jpeg?download=true"</span>

<span class="hljs-keyword">with</span> urllib.request.urlopen(url) <span class="hljs-keyword">as</span> url_response:
    img_array = np.array(<span class="hljs-built_in">bytearray</span>(url_response.read()), dtype=np.uint8)

original = cv2.imdecode(img_array, -<span class="hljs-number">1</span>)
image = image_resize(original)
expand_pixels_to_square = <span class="hljs-number">1024</span> - image.shape[<span class="hljs-number">1</span>]  <span class="hljs-comment"># image.shape[1] for horizontal, image.shape[0] for vertical</span>
image, mask = process_image(
    image, expand_pixels=expand_pixels_to_square, direction=direction, inpaint_mask_color=inpaint_mask_color
)

ip_adapter_image = []
<span class="hljs-keyword">for</span> index, part <span class="hljs-keyword">in</span> <span class="hljs-built_in">enumerate</span>(slice_image(original)):
    ip_adapter_image.append(part)

generated = generate_image(prompt, negative_prompt, image, mask, ip_adapter_image)
final_image = generated

<span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(times_to_expand):
    image, mask = process_image(
        final_image, direction=direction, expand_pixels=expand_pixels, inpaint_mask_color=inpaint_mask_color
    )

    ip_adapter_image = []
    <span class="hljs-keyword">for</span> index, part <span class="hljs-keyword">in</span> <span class="hljs-built_in">enumerate</span>(slice_image(generated)):
        ip_adapter_image.append(part)

    generated = generate_image(prompt, negative_prompt, image, mask, ip_adapter_image)
    final_image = merge_images(final_image, generated, <span class="hljs-number">256</span>, direction)

cv2.imwrite(<span class="hljs-string">"result.png"</span>, final_image)
</code></pre>
