Use OpenAI's CLIP for style transfer
Please read the topic categorydescriptionto understand what this is all aboutDescriptionOne of the most exciting developments in 2021 was the release of OpenAI’s CLIP model, which was trained on a variety of (text, image) pairs. One of the cool things you can do with this model is use it to combine text and image embeddings to performneural style transfer. In neural style transfer, the idea is to provide a prompt like “a starry night painting” and an image, and then get the model to produce a painting of the image in that style.The goal of this project is to learn whether CLIP can produce good paintings from text prompts.Model(s)TheCLIP modelscan be found on the HubDatasetsFor this project, you probably won’t need an actual dataset to perform neural style transfer. Just a single image should be enough to tune CLIP and an image encoder. Of course, you are free to experiment with larger datsets if you want!ChallengesThis project goes beyond that concepts introduced in Part II of theCourse, so some familiarity with computer vision would be useful. Having said that, theTransformers API is similar for image tasks, so if you know how thepipeline()function works, then you’ll have no trouble adapting to this new domain.Desired project outcomesCreate a Streamlit or Gradio app onSpacesthat allows a user to provide an image and a text prompt, and produces a painting of that image in the desired styleAdditional resourcesYou can Google “neural style transfer” to find plenty of information about this technique. Here one advanced example to give you an idea:GitHub - orpatashnik/StyleCLIP: Official Implementation for "StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery" (ICCV 2021 Oral)Discord channelTo chat and organise with other people interested in this project, head over to ourDiscordand:Follow the instructions on the#join-coursechannelJoin the#neural-style-transferchannelJust make sure you comment here to indicate that you’ll be contributing to this project
2021-11-15T22:38:17Z
Very interesting topic, I am in!
2021-11-16T01:45:49Z
Hi!I’d also like to contribute to this project. I believe it’s quite aligned with another project I’m working on, namely “Image search using CLIP”.
2021-11-16T11:11:45Z
lewtun:neural-style-Hi, I would like to contribute to this project