AutoencoderKL.scaling_factor and VaeImageProcessor
While working onan exampleof using AutoencoderKL and AutoencoderTiny (TAESD), I stumbled over the use ofAutoencoderKL.scaling_factor.It’s some factor that is necessary for using the VAE with existing Stable Diffusion models, but is not applied by any of the class’s methods, nor by theVAE Image Processorclass?In the past, I’ve used theStableDiffusionPipeline#decode_latentsmethod to do that, but that method now emits this warning:The decode_latents method is deprecated and will be removed in a future version. Please use VaeImageProcessor instead.How am I to replace my usage of this deprecated method?
2023-08-20T16:45:47Z
Diffusers appliesvae.config.scaling_factormanuallywithin the image generation pipelines, rather than within the SD VAE itself. If you are working with the Diffusers SD VAE module you’ll need to do the same thing. Examples:github.comhuggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_latent_upscale.py#L387# 3. Encode input prompttext_embeddings, text_pooler_out = self._encode_prompt(prompt, device, do_classifier_free_guidance, negative_prompt)# 4. Preprocess imageimage = self.image_processor.preprocess(image)image = image.to(dtype=text_embeddings.dtype, device=device)if image.shape[1] == 3:# encode image if not in latent-space yetimage = self.vae.encode(image).latent_dist.sample() * self.vae.config.scaling_factor# 5. set timestepsself.scheduler.set_timesteps(num_inference_steps, device=device)timesteps = self.scheduler.timestepsbatch_multiplier = 2 if do_classifier_free_guidance else 1image = image[None, :] if image.ndim == 3 else imageimage = torch.cat([image] * batch_multiplier)# 5. Add noise to image (set to be 0):github.comhuggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_latent_upscale.py#L485# compute the previous noisy sample x_t -> x_t-1latents = self.scheduler.step(noise_pred, t, latents).prev_sample# call the callback, if providedif i == len(timesteps) - 1 or ((i + 1) > num_warmup_steps and (i + 1) % self.scheduler.order == 0):progress_bar.update()if callback is not None and i % callback_steps == 0:callback(i, t, latents)if not output_type == "latent":image = self.vae.decode(latents / self.vae.config.scaling_factor, return_dict=False)[0]else:image = latentsimage = self.image_processor.postprocess(image, output_type=output_type)if not return_dict:return (image,)return ImagePipelineOutput(images=image)
2023-08-20T17:20:12Z
So the deprecation message is a misdirect, and the class it references does not replace it?In that case, why deprecate thedecode_latentsmethod at all?madebyollin:Diffusers appliesvae.config.scaling_factormanuallywithin the image generation pipelines,Is there ever a time when one would callvae.encodeorvae.decodewithoutapplying this scaling factor?
2023-08-20T17:37:52Z
So the deprecation message is a misdirect, and the class it references does not replace it?Kinda.decode_latentswas doing several things:Applying thescaling_factorRunning the VAE decoderRescaling decoded outputs from [-1, 1] → [0, 1] and clampingConverting tensor format from NCHW torch GPU to NHWC numpy CPU.TheVaeImageProcessoronly doessteps 3 and 4of that, so it’s a partial replacement.Is there ever a time when one would callvae.encodeorvae.decodewithoutapplying this scaling factor?I’m not aware of any (this is why TAESD does not have ascaling_factor). I suspect the separatescaling_factoris inherited from Stability / CompVis code where they treat thescaling_factoras aparameter of the diffusion model(i.e. “how much did we scale the VAE latents down when training this particular diffusion model?”) rather than as a property of the VAE.
2023-08-23T00:03:30Z
Right. Given that (a) misleading deprecation warnings are bad, and (b) methods that never return the correct values are bad, I request that we eitherComplete the migration of the VAEscaling_factoraway from the Pipeline class and in to the Autoencoder class. Always apply it to the values leavingencodeand going in todecode. Or,Provide methods wrapping the autoencoder that do the necessary thing. This could mean keepingdecode_latents(and a correspondingencode) method on the Pipeline, or it could mean adding that scaling to the VaeImageProcessor’s preprocessing and postprocessing methods.What’ll it be?
2023-08-28T15:16:39Z
It looks like the “do scaling in encode / decode” approach was already proposed in [[wip] init scale_value on vae by williamberman · Pull Request #1515 · huggingface/diffusers · GitHub] but rejected in favor of [make scaling factor a config arg of vae/vqvae by patil-suraj · Pull Request #1860 · huggingface/diffusers · GitHub] due to backwards compatibility concerns (since lots of external code wasalreadyapplying the scaling factor outside the VAE).The “create new encode / decode methods that also scale / unscale, and use those wherever possible” approach (alsomentioned- but not implemented) sounds like a good idea to me. It’s plausible that the diffusers team would accept a PR doing just that. But it’s not my call (I’m just some random dev)
2023-08-29T19:01:20Z
Thank you Ollin! I’ve been hoping to draw out the maintainers whodomake the API decisions, but I must have posted during summer vacation. Having the links to the previous discussions is handy though.