
# KerasCV Stable Diffusion in Diffusers 🧨🤗

DreamBooth model for the `puggieace` concept trained by nielsgl on the `nielsgl/dreambooth-ace` dataset.
It can be used by modifying the `instance_prompt`: **a photo of puggieace**.

The examples are from 2 different Keras CV models (`StableDiffusion` and `StableDiffusionV2`, corresponding to Stable Diffusion V1.4 and V2.1, respectively) trained on the same dataset (`nielsgl/dreambooth-ace`).

## Description

The Stable Diffusion V2 pipeline contained in the corresponding repository (`nielsgl/dreambooth-keras-pug-ace-sd2.1`) was created using a modified version of [this Space](https://huggingface.co/spaces/sayakpaul/convert-kerascv-sd-diffusers) for StableDiffusionV2 from KerasCV. The purpose is to convert the KerasCV Stable Diffusion weights in a way that is compatible with [Diffusers](https://github.com/huggingface/diffusers). This allows users to fine-tune using KerasCV and use the fine-tuned weights in Diffusers taking advantage of its nifty features (like [schedulers](https://huggingface.co/docs/diffusers/main/en/using-diffusers/schedulers), [fast attention](https://huggingface.co/docs/diffusers/optimization/fp16), etc.).
This model was created as part of the Keras DreamBooth Sprint 🔥. Visit the [organisation page](https://huggingface.co/keras-dreambooth) for instructions on how to take part!

## Examples

### Stable Diffusion V1.4

> Portrait of puggieace dog as a Roman Emperor, city in background

![Portrait of puggieace dog as a Roman Emperor, city in background, ultra realistic, intricate details, eerie, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater and magali villeneuve and alphonse mucha](examples/emperor-1.4.jpeg)

> Photo of puggieace dog wearing sunglasses on the beach, sunset in background, golden hour

![Photo of puggieace dog wearing sunglasses on the beach, sunset in background, golden hour](examples/beach-1.4.jpg)

> Photo of cute puggieace dog as an astronaut, planet and spaceship in background

![Photo of cute puggieace dog as an astronaut, planet and spaceship in background, ultra realistic, intricate details, highly detailed, photorealistic, octane render, 8 k, unreal engine. trending on artstation](examples/astronaut-1.4.jpg)

### Stable Diffusion V2.1

> Portrait painting of a cute puggieace dog as a samurai

![Portrait painting of a cute puggieace dog as a samurai, ultra realistic, concept art, intricate details, eerie, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater and magali villeneuve and alphonse mucha](examples/samurai-2.1.jpg)

> Photo of cute puggieace dog as an astronaut, space and planet in background

![Photo of cute puggieace dog as an astronaut, space and planet in background, ultra realistic, concept art, intricate details, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater, trending on artstation](examples/astronaut-2.1.jpg)

> A photo of a cute puggieace dog getting a haircut in a barbershop

![A photo of a cute puggieace dog getting a haircut in a barbershop, ultra realistic, intricate details, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater and magali villeneuve and alphonse mucha](examples/haircut-2.1.jpg)

> Portrait photo of puggieace dog in New York

![Portrait photo of puggieace dog in New York, city and skyscrapers in background, highly detailed, photorealistic, hdr, 4k](examples/ny-2.1.jpg)

> Portrait of puggieace dog as a Roman Emperor, city in background

![Portrait of puggieace dog as a Roman Emperor, city in background, ultra realistic, intricate details, eerie, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater and magali villeneuve and alphonse mucha](examples/emperor-2.1.jpg)


## Usage with Stable Diffusion V1.4

```python
from huggingface_hub import from_pretrained_keras
import keras_cv
import matplotlib.pyplot as plt


model = keras_cv.models.StableDiffusion(img_width=512, img_height=512, jit_compile=True)
model._diffusion_model = from_pretrained_keras("nielsgl/dreambooth-pug-ace")
model._text_encoder = from_pretrained_keras("nielsgl/dreambooth-pug-ace-text-encoder")

images = model.text_to_image("a photo of puggieace dog on the beach", batch_size=3)
plt.imshow(image[0])
```

## Usage with Stable Diffusion V2.1

```python
from diffusers import StableDiffusionPipeline

pipeline = StableDiffusionPipeline.from_pretrained('nielsgl/dreambooth-keras-pug-ace-sd2.1')
image = pipeline().images[0]
image
```

### Training hyperparameters

The following hyperparameters were used during training for Stable Diffusion v1.4:

| Hyperparameters | Value |
| :-- | :-- |
| name | RMSprop |
| weight_decay | None |
| clipnorm | None |
| global_clipnorm | None |
| clipvalue | None |
| use_ema | False |
| ema_momentum | 0.99 |
| ema_overwrite_frequency | 100 |
| jit_compile | True |
| is_legacy_optimizer | False |
| learning_rate | 0.0010000000474974513 |
| rho | 0.9 |
| momentum | 0.0 |
| epsilon | 1e-07 |
| centered | False |
| training_precision | float32 |
{"license": "creativeml-openrail-m", "library_name": "keras", "tags": ["keras", "diffusers", "stable-diffusion", "text-to-image", "diffusion-models-class", "keras-dreambooth", "nature"], "datasets": ["nielsgl/dreambooth-ace"], "inference": true, "widget": [{"text": "a photo of puggieace dog on the beach, sunset in background"}], "pipeline_tag": "text-to-image", "emoji": "\ud83d\udc36"}
keras diffusers stable-diffusion text-to-image diffusion-models-class keras-dreambooth nature dataset:nielsgl/dreambooth-ace license:creativeml-openrail-m region:us
TAGS
#keras #diffusers #stable-diffusion #text-to-image #diffusion-models-class #keras-dreambooth #nature #dataset-nielsgl/dreambooth-ace #license-creativeml-openrail-m #region-us 

KerasCV Stable Diffusion in Diffusers
=====================================


DreamBooth model for the 'puggieace' concept trained by nielsgl on the 'nielsgl/dreambooth-ace' dataset.
It can be used by modifying the 'instance\_prompt': a photo of puggieace.


The examples are from 2 different Keras CV models ('StableDiffusion' and 'StableDiffusionV2', corresponding to Stable Diffusion V1.4 and V2.1, respectively) trained on the same dataset ('nielsgl/dreambooth-ace').


Description
-----------


The Stable Diffusion V2 pipeline contained in the corresponding repository ('nielsgl/dreambooth-keras-pug-ace-sd2.1') was created using a modified version of this Space for StableDiffusionV2 from KerasCV. The purpose is to convert the KerasCV Stable Diffusion weights in a way that is compatible with Diffusers. This allows users to fine-tune using KerasCV and use the fine-tuned weights in Diffusers taking advantage of its nifty features (like schedulers, fast attention, etc.).
This model was created as part of the Keras DreamBooth Sprint . Visit the organisation page for instructions on how to take part!


Examples
--------


### Stable Diffusion V1.4



> 
> Portrait of puggieace dog as a Roman Emperor, city in background
> 
> 
> 


!Portrait of puggieace dog as a Roman Emperor, city in background, ultra realistic, intricate details, eerie, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater and magali villeneuve and alphonse mucha



> 
> Photo of puggieace dog wearing sunglasses on the beach, sunset in background, golden hour
> 
> 
> 


!Photo of puggieace dog wearing sunglasses on the beach, sunset in background, golden hour



> 
> Photo of cute puggieace dog as an astronaut, planet and spaceship in background
> 
> 
> 


!Photo of cute puggieace dog as an astronaut, planet and spaceship in background, ultra realistic, intricate details, highly detailed, photorealistic, octane render, 8 k, unreal engine. trending on artstation


### Stable Diffusion V2.1



> 
> Portrait painting of a cute puggieace dog as a samurai
> 
> 
> 


!Portrait painting of a cute puggieace dog as a samurai, ultra realistic, concept art, intricate details, eerie, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater and magali villeneuve and alphonse mucha



> 
> Photo of cute puggieace dog as an astronaut, space and planet in background
> 
> 
> 


!Photo of cute puggieace dog as an astronaut, space and planet in background, ultra realistic, concept art, intricate details, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater, trending on artstation



> 
> A photo of a cute puggieace dog getting a haircut in a barbershop
> 
> 
> 


!A photo of a cute puggieace dog getting a haircut in a barbershop, ultra realistic, intricate details, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater and magali villeneuve and alphonse mucha



> 
> Portrait photo of puggieace dog in New York
> 
> 
> 


!Portrait photo of puggieace dog in New York, city and skyscrapers in background, highly detailed, photorealistic, hdr, 4k



> 
> Portrait of puggieace dog as a Roman Emperor, city in background
> 
> 
> 


!Portrait of puggieace dog as a Roman Emperor, city in background, ultra realistic, intricate details, eerie, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater and magali villeneuve and alphonse mucha


Usage with Stable Diffusion V1.4
--------------------------------


Usage with Stable Diffusion V2.1
--------------------------------


### Training hyperparameters


The following hyperparameters were used during training for Stable Diffusion v1.4:




### Stable Diffusion V1.4



> 
> Portrait of puggieace dog as a Roman Emperor, city in background
> 
> 
> 


!Portrait of puggieace dog as a Roman Emperor, city in background, ultra realistic, intricate details, eerie, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater and magali villeneuve and alphonse mucha



> 
> Photo of puggieace dog wearing sunglasses on the beach, sunset in background, golden hour
> 
> 
> 


!Photo of puggieace dog wearing sunglasses on the beach, sunset in background, golden hour



> 
> Photo of cute puggieace dog as an astronaut, planet and spaceship in background
> 
> 
> 


!Photo of cute puggieace dog as an astronaut, planet and spaceship in background, ultra realistic, intricate details, highly detailed, photorealistic, octane render, 8 k, unreal engine. trending on artstation ### Stable Diffusion V2.1



> 
> Portrait painting of a cute puggieace dog as a samurai
> 
> 
> 


!Portrait painting of a cute puggieace dog as a samurai, ultra realistic, concept art, intricate details, eerie, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater and magali villeneuve and alphonse mucha



> 
> Photo of cute puggieace dog as an astronaut, space and planet in background
> 
> 
> 


!Photo of cute puggieace dog as an astronaut, space and planet in background, ultra realistic, concept art, intricate details, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater, trending on artstation



> 
> A photo of a cute puggieace dog getting a haircut in a barbershop
> 
> 
> 


!A photo of a cute puggieace dog getting a haircut in a barbershop, ultra realistic, intricate details, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater and magali villeneuve and alphonse mucha



> 
> Portrait photo of puggieace dog in New York
> 
> 
> 


!Portrait photo of puggieace dog in New York, city and skyscrapers in background, highly detailed, photorealistic, hdr, 4k



> 
> Portrait of puggieace dog as a Roman Emperor, city in background
> 
> 
> 


!Portrait of puggieace dog as a Roman Emperor, city in background, ultra realistic, intricate details, eerie, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater and magali villeneuve and alphonse mucha


Usage with Stable Diffusion V1.4
--------------------------------


Usage with Stable Diffusion V2.1
-------------------------------- ### Training hyperparameters


The following hyperparameters were used during training for Stable Diffusion v1.4:
TAGS
#keras #diffusers #stable-diffusion #text-to-image #diffusion-models-class #keras-dreambooth #nature #dataset-nielsgl/dreambooth-ace #license-creativeml-openrail-m #region-us 
 ### Stable Diffusion V1.4



> 
> Portrait of puggieace dog as a Roman Emperor, city in background
> 
> 
> 


!Portrait of puggieace dog as a Roman Emperor, city in background, ultra realistic, intricate details, eerie, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater and magali villeneuve and alphonse mucha



> 
> Photo of puggieace dog wearing sunglasses on the beach, sunset in background, golden hour
> 
> 
> 


!Photo of puggieace dog wearing sunglasses on the beach, sunset in background, golden hour



> 
> Photo of cute puggieace dog as an astronaut, planet and spaceship in background
> 
> 
> 


!Photo of cute puggieace dog as an astronaut, planet and spaceship in background, ultra realistic, intricate details, highly detailed, photorealistic, octane render, 8 k, unreal engine. trending on artstation ### Stable Diffusion V2.1



> 
> Portrait painting of a cute puggieace dog as a samurai
> 
> 
> 


!Portrait painting of a cute puggieace dog as a samurai, ultra realistic, concept art, intricate details, eerie, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater and magali villeneuve and alphonse mucha



> 
> Photo of cute puggieace dog as an astronaut, space and planet in background
> 
> 
> 


!Photo of cute puggieace dog as an astronaut, space and planet in background, ultra realistic, concept art, intricate details, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater, trending on artstation



> 
> A photo of a cute puggieace dog getting a haircut in a barbershop
> 
> 
> 


!A photo of a cute puggieace dog getting a haircut in a barbershop, ultra realistic, intricate details, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater and magali villeneuve and alphonse mucha



> 
> Portrait photo of puggieace dog in New York
> 
> 
> 


!Portrait photo of puggieace dog in New York, city and skyscrapers in background, highly detailed, photorealistic, hdr, 4k



> 
> Portrait of puggieace dog as a Roman Emperor, city in background
> 
> 
> 


!Portrait of puggieace dog as a Roman Emperor, city in background, ultra realistic, intricate details, eerie, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater and magali villeneuve and alphonse mucha


Usage with Stable Diffusion V1.4
--------------------------------


Usage with Stable Diffusion V2.1
-------------------------------- ### Training hyperparameters


The following hyperparameters were used during training for Stable Diffusion v1.4:
TAGS
#keras #diffusers #stable-diffusion #text-to-image #diffusion-models-class #keras-dreambooth #nature #dataset-nielsgl/dreambooth-ace #license-creativeml-openrail-m #region-us 
### Stable Diffusion V1.4



> 
> Portrait of puggieace dog as a Roman Emperor, city in background
> 
> 
> 


!Portrait of puggieace dog as a Roman Emperor, city in background, ultra realistic, intricate details, eerie, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater and magali villeneuve and alphonse mucha



> 
> Photo of puggieace dog wearing sunglasses on the beach, sunset in background, golden hour
> 
> 
> 


!Photo of puggieace dog wearing sunglasses on the beach, sunset in background, golden hour



> 
> Photo of cute puggieace dog as an astronaut, planet and spaceship in background
> 
> 
> 


!Photo of cute puggieace dog as an astronaut, planet and spaceship in background, ultra realistic, intricate details, highly detailed, photorealistic, octane render, 8 k, unreal engine. trending on artstation### Stable Diffusion V2.1



> 
> Portrait painting of a cute puggieace dog as a samurai
> 
> 
> 


!Portrait painting of a cute puggieace dog as a samurai, ultra realistic, concept art, intricate details, eerie, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater and magali villeneuve and alphonse mucha



> 
> Photo of cute puggieace dog as an astronaut, space and planet in background
> 
> 
> 


!Photo of cute puggieace dog as an astronaut, space and planet in background, ultra realistic, concept art, intricate details, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater, trending on artstation



> 
> A photo of a cute puggieace dog getting a haircut in a barbershop
> 
> 
> 


!A photo of a cute puggieace dog getting a haircut in a barbershop, ultra realistic, intricate details, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater and magali villeneuve and alphonse mucha



> 
> Portrait photo of puggieace dog in New York
> 
> 
> 


!Portrait photo of puggieace dog in New York, city and skyscrapers in background, highly detailed, photorealistic, hdr, 4k



> 
> Portrait of puggieace dog as a Roman Emperor, city in background
> 
> 
> 


!Portrait of puggieace dog as a Roman Emperor, city in background, ultra realistic, intricate details, eerie, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater and magali villeneuve and alphonse mucha


Usage with Stable Diffusion V1.4
--------------------------------


Usage with Stable Diffusion V2.1
--------------------------------### Training hyperparameters


The following hyperparameters were used during training for Stable Diffusion v1.4: