Uncovering the Disentanglement Capability in Text-to-Image Diffusion Models

Generative models have been widely studied in computer vision. Recently, diffusion models have drawn substantial attention due to the high quality of their generated images. A key desired property of image generative models is the ability to disentangle different attributes, which should enable modiﬁcation towards a style without changing the semantic content, and the modiﬁcation parameters should generalize to different images. Previous studies have found that generative adversarial networks (GANs) are inherently endowed with such disentanglement capability, so they can perform disentangled image editing without re-training or ﬁne-tuning the network. In this work, we explore whether diffusion models are also inherently equipped with such a capability. Our ﬁnding is that for stable diffusion models, by partially changing the input text embedding from a neutral description ( e.g., “a photo of person”) to one with style ( e.g., “a photo of person with smile”) while ﬁxing all the Gaussian random noises introduced during the denoising process, the generated images can be modiﬁed towards the target style without changing the semantic content. Based on this ﬁnding, we further propose a simple, light-weight image editing algorithm where the mixing weights of the two text embeddings are optimized for style matching and content preservation. This entire process only involves optimizing over around 50 parameters and does not ﬁne-tune the diffusion model itself. Experiments show that the proposed method can modify a wide range of attributes, with the performance outperforming diffusion-model-based image-editing algorithms that require ﬁne-tuning. The optimized weights generalize well to different images. Our code is publicly available at https://github.com/UCSB-NLP-Chang/DiffusionDisentanglement .