Pass additional information into Key and Value weights of Stable Diffusion
Hello everyone, I’m working on integrating some code to work with Stable Diffusion. The code is found here (GitHub - lucidrains/perfusion-pytorch: Implementation of Key-Locked Rank One Editing, from Nvidia AI). What I need to do is to replace the Key and Value weights inside each cross attention block within Stable Diffusion. Essentially what it does is wrap each Key Value matrix and do some additional calculations and return the output. This is extremely close to the Custom Diffusion process that you can findhere. However, the code I’m using requires additional parameters to be passed in during training/inference. What I would like to do is to pass in the additional information in thecall()method of the UNet2DConditionModel class, and then have that be passed into the wrapper. What I’ve thought of so far is using a customAttnProcessorclass, similar to theCustomDiffusionAttnProcessorclass. However I can’t figure out how to use these classes to accept the additional parameters that would be passed into the UNet2DConditionModel. Any help would be greatly appreciated.
2024-01-29T08:42:57Z
Hey Irow, i am working on the same project let me know if we can contact through mail or or messages on huggingface.