It's raining diffusion personalization techniques☔️🎭🖼️

<p>

Recently, generating high quality portraits from refrence photos was made possible with as little as a single reference image &amp; without any optimization⚡️</p>
<p><a href="https://cdn-uploads.huggingface.co/production/uploads/638f308fc4444c6ca870b60a/U4Jxr5L5htNaxv02CRiNR.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/638f308fc4444c6ca870b60a/U4Jxr5L5htNaxv02CRiNR.png"/></a></p>
<p><em>figure taken from <a href="https://huggingface.co/papers/2401.07519">InstantID: Zero-shot Identity-Preserving Generation in Seconds</a></em></p>
<p>Using these new zero-shot methods, one can easily generate a self portrait with their choice of style, composition, and background👩🏻‍🎨</p>
<p>Here are 3 zero-shot pipelines to know and try🚀 </p>
<ol>
<li><a href="https://huggingface.co/spaces/multimodalart/face-to-all">Face-to-all</a></li>
<li><a href="https://huggingface.co/spaces/InstantX/InstantID">InstantID</a></li>
<li><a href="https://huggingface.co/spaces/multimodalart/Ip-Adapter-FaceID">IP Adapter FaceID</a></li>
</ol>
<p><a href="https://huggingface.co/spaces/multimodalart/Ip-Adapter-FaceID">🎭<strong>IP Adapter FaceID</strong>🎭</a></p>
<ul>
<li><a href="https://huggingface.co/papers/2308.06721">📗IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models</a></li>
<li><a href="https://github.com/tencent-ailab/IP-Adapter" rel="nofollow">Code 👩‍💻</a></li>
<li><a href="https://huggingface.co/spaces/multimodalart/Ip-Adapter-FaceID">Demo 🤗</a></li>
</ul>
<p>IP Adapters consist of 2 core components:</p>
<p><a href="https://cdn-uploads.huggingface.co/production/uploads/638f308fc4444c6ca870b60a/AAFuso-fvot29zyU4GQNc.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/638f308fc4444c6ca870b60a/AAFuso-fvot29zyU4GQNc.png"/></a></p>
<ol>
<li>An <code>image encoder</code> to extract image features (from the reference image/s)</li>
<li><strong>Decoupled cross-attention</strong> layers for text features and image features. A new cross-attention layer is added for each cross-attention layer in the original UNet model to insert image features.
💡To improve face fidelity, in <a href="https://huggingface.co/h94/IP-Adapter-FaceID">IP Adapter FaceID</a>, face embeddings were introduced, instead of (or in addition to in IP Adapter FaceID Plus) to CLIP embeddings.</li>
</ol>
<p><a href="https://huggingface.co/spaces/InstantX/InstantID">🎭<strong>InstantID</strong>🎭</a></p>
<ul>
<li><a href="https://huggingface.co/papers/2401.07519">📘InstantID: Zero-shot Identity-Preserving Generation in Seconds</a></li>
<li><a href="https://github.com/InstantID/InstantID" rel="nofollow">Code 👩‍💻</a></li>
<li><a href="https://huggingface.co/spaces/InstantX/InstantID">Demo 🤗</a></li>
</ul>
<p><a href="https://cdn-uploads.huggingface.co/production/uploads/638f308fc4444c6ca870b60a/DdWnrxg-FC2PkDr5C3zYH.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/638f308fc4444c6ca870b60a/DdWnrxg-FC2PkDr5C3zYH.png"/></a></p>
<p>Similar to IP Adapter, InstantID also makes use of id embeddings and decoupled cross attention, but adds a new component: <strong>Identity Net</strong></p>
<p>💡IdentityNet - an adapted ControlNet - meant to encode the detailed features from the reference facial image with additional spatial control, with 2 main modifications to ControlNet:</p>
<p>❶ Instead of fine-grained OpenPose facial keypoints, only five facial keypoints are used (two for the eyes, one for the nose, and two for the mouth) for conditional input. </p>
<p>❷ Eliminate text prompts and use ID embedding as conditions for cross-attention layers in the ControlNet</p>
<p><a href="https://huggingface.co/spaces/multimodalart/face-to-all">🎭<strong>Face-to-all</strong>🎭</a></p>
<ul>
<li><a href="https://huggingface.co/spaces/multimodalart/face-to-all/tree/main">Code 👩‍💻</a></li>
<li><a href="https://huggingface.co/spaces/multimodalart/face-to-all">Demo 🤗</a></li>
</ul>
<p>a diffusers 🧨 workflow inspired by <a href="https://huggingface.co/fofr">@fofr</a> Face-to-Many ComfyUI workflow🔥
<a href="https://cdn-uploads.huggingface.co/production/uploads/638f308fc4444c6ca870b60a/Q8rv4zS0MGi5uXx7jFN4Y.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/638f308fc4444c6ca870b60a/Q8rv4zS0MGi5uXx7jFN4Y.png"/></a></p>
<p>This workflow extends the original InstantID pipeline &amp; combines it with <em>any</em> SDXL LoRA:</p>
<ol>
<li>adding the option to <strong>stylize with all style sdxl LoRAs</strong> - especially useful for styles that aren't known to the base diffusion model (browse the <a href="https://huggingface.co/spaces/enzostvs/lora-studio">LoRA Studio</a> for inspo ✨)</li>
<li>improving structure preservation - maintaining the composition of the reference image.</li>
</ol>
