text
I'm happy to announce that ✨ Image to Music v2 ✨ is ready for you to try and i hope you'll like it too ! 😌
I'm happy to announce that ✨ Image to Music v2 ✨ is ready for you to try and i hope you'll like it too ! 😌
new_line
new_line
text
This new version has been crafted with transparency in mind, 
This new version has been crafted with transparency in mind, 
new_line
text
so you can understand the process of translating an image to a musical equivalent. 
so you can understand the process of translating an image to a musical equivalent. 
new_line
new_line
text
How does it works under the hood ? 🤔
How does it works under the hood ? 🤔
new_line
new_line
text
First, we get a very literal caption from 
First, we get a very literal caption from 
resource
https://huggingface.co/microsoft/kosmos-2-patch14-224
model
microsoft/kosmos-2-patch14-224
https://huggingface.co/microsoft/kosmos-2-patch14-224
text
; this caption is then given to a LLM Agent (currently 
; this caption is then given to a LLM Agent (currently 
resource
https://huggingface.co/HuggingFaceH4/zephyr-7b-beta
model
HuggingFaceH4/zephyr-7b-beta
https://huggingface.co/HuggingFaceH4/zephyr-7b-beta
text
 )which task is to translate the image caption to a musical and inspirational prompt for the next step.
 )which task is to translate the image caption to a musical and inspirational prompt for the next step.
new_line
new_line
text
Once we got a nice musical text from the LLM, we can send it to the text-to-music model of your choice: 
Once we got a nice musical text from the LLM, we can send it to the text-to-music model of your choice: 
new_line
text
MAGNet, MusicGen, AudioLDM-2, Riffusion or Mustango
MAGNet, MusicGen, AudioLDM-2, Riffusion or Mustango
new_line
new_line
text
Instead of the previous version of Image to Music which used Mubert API, and could output curious and obscure combinations, we only provide open sourced models available on the hub, called via the gradio API.
Instead of the previous version of Image to Music which used Mubert API, and could output curious and obscure combinations, we only provide open sourced models available on the hub, called via the gradio API.
new_line
new_line
text
Also i guess the music result should be more accurate to the atmosphere of the image input, thanks to the LLM Agent step. 
Also i guess the music result should be more accurate to the atmosphere of the image input, thanks to the LLM Agent step. 
new_line
new_line
text
Pro tip, you can adjust the inspirational prompt to match your expectations, according to the chosen model and specific behavior of each one 👌
Pro tip, you can adjust the inspirational prompt to match your expectations, according to the chosen model and specific behavior of each one 👌
new_line
new_line
text
Try it, explore different models and tell me which one is your favorite 🤗
Try it, explore different models and tell me which one is your favorite 🤗
new_line
text
—› 
—› 
resource
https://huggingface.co/spaces/fffiloni/image-to-music-v2
space
fffiloni/image-to-music-v2
https://huggingface.co/spaces/fffiloni/image-to-music-v2
new_line
I'm happy to announce that ✨ Image to Music v2 ✨ is ready for you to try and i hope you'll like it too ! 😌

This new version has been crafted with transparency in mind, 
so you can understand the process of translating an image to a musical equivalent. 

How does it works under the hood ? 🤔

First, we get a very literal caption from https://huggingface.co/microsoft/kosmos-2-patch14-224; this caption is then given to a LLM Agent (currently https://huggingface.co/HuggingFaceH4/zephyr-7b-beta )which task is to translate the image caption to a musical and inspirational prompt for the next step.

Once we got a nice musical text from the LLM, we can send it to the text-to-music model of your choice: 
MAGNet, MusicGen, AudioLDM-2, Riffusion or Mustango

Instead of the previous version of Image to Music which used Mubert API, and could output curious and obscure combinations, we only provide open sourced models available on the hub, called via the gradio API.

Also i guess the music result should be more accurate to the atmosphere of the image input, thanks to the LLM Agent step. 

Pro tip, you can adjust the inspirational prompt to match your expectations, according to the chosen model and specific behavior of each one 👌

Try it, explore different models and tell me which one is your favorite 🤗
—› https://huggingface.co/spaces/fffiloni/image-to-music-v2

❤️
victor samusenps osanseviero phanes abidlabs koheingt adhisetiawan Indomitable-ai boqsc deepkyu mvaloatto MaziyarPanahi powzone kelsy Chief-Inspector pe65374 Swhalen98 aceeee Aixbox
👍
naorex ivied7 MehdiLeZ eisneim OmbelineM
🤗
Bssayla
🚀
Aixbox
https://cdn-avatars.huggingface.co/v1/production/uploads/63893d4c184615e463aa24b8/S1flsX_26OF6ZJBVcPlaf.jpeg
Matt Valoatto
mvaloatto
user
https://cdn-avatars.huggingface.co/v1/production/uploads/5f17f0a0925b9863e28ad517/X7QKoiXbUtEZSG9jyvfk3.jpeg
Victor Mustar
victor
user
https://cdn-avatars.huggingface.co/v1/production/uploads/5fd5e18a90b6dc4633f6d292/gZXHW5dd9R86AV9LMZ--y.png
Maziyar Panahi
MaziyarPanahi
user
https://cdn-avatars.huggingface.co/v1/production/uploads/61868ce808aae0b5499a2a95/F6BA0anbsoY_Z7M1JrwOe.jpeg
Sylvain Filoni
fffiloni
user
/avatars/9f18cf580ec917926be976fc061d00bd.svg
Andrew Glyadchenko
andgly95
user
https://cdn-avatars.huggingface.co/v1/production/uploads/1677620936417-63fe761688b9695964bf2696.jpeg
paolo calveri
acebruck
user
https://cdn-avatars.huggingface.co/v1/production/uploads/6544b4a04200ae379ea69302/-OnV6YPGaTUyBKOJiq4WH.png
Mehdi Zatar
MehdiLeZ
user