Generate GIF reply to English text with VQGAN + CLIP
So this would be a hybrid dialog system combining intent recognition with text-to-image generation. The idea is to train the model to generate visual replies (static or dynamic GIFs, depending on how hard the latter is) to English-language text input. E.g.:text input: ‘this restaurant is the bomb’model output: generated picture of exploding restaurantThere are quite a few GIF and text-to-image datasets out there, but we’d need to combine them in a way that the model can be trained. A large part of the work will be to implement VQGAN and CLIP in JAX.
2021-06-24T05:14:26Z
I’m also interested in this taskI would like to join forces if you are interested. I’m pretty confident in NLP.I know CLIP, (some of) vqgan model but don’t know Jax.My github:cceydaAlso see thisdataset
2021-06-24T08:15:44Z
This sounds really cool!CLIP is already available in JAXtransformers/modeling_flax_clip.py at master · huggingface/transformers · GitHub
2021-06-24T09:55:50Z
This sounds great!We’re also working in atext-to-imagewidget so it could be fun to try it out with the proposed projecthttps://github.com/huggingface/huggingface_hub/issues/113
2021-06-24T12:53:49Z
BTW I think It would be fun to try text → emoji/emoticon (image not unicode)If we think about all the different renderings of emoticons between platforms there is reasonable variation to generate the ultimate DeepEmoji set. Would be an easier starting point than reaction gifs…ooh could also animate emojis…Also
2021-06-24T13:20:18Z
This proposed project is interesting.My interest is Computer Vision, but have no experience with JAX so far.Anyways, I love to learn and work on a project that relates NLP with Computer Vision tasks.So I would like to join this project as well.My time zone is Tokyo (GMT+9)
2021-06-24T16:08:45Z
You might want to check outjulien-c/reactiongif · Datasets at Hugging Face. It could be useful
2021-06-24T16:41:13Z
It looks like the version on datasets is missing the latest gif files updateCode to download gifs? · Issue #1 · bshmueli/ReactionGIF · GitHubBut no worries I already grabbed the files & I can make a PR to add those files if there are no license issues… but I’m not sure
2021-06-24T22:51:53Z
I like this idea a lot and would be interested in joining!
2021-06-25T03:05:06Z
very interesting, I would love to contribute in this project. I have experience in CV as well as in NLP.
2021-06-25T09:06:47Z
hi everyone, thanks for your replies!I’m very busy at work at the moment, I’ll have a look at the suggestions over the weekend…Any suggestions on collaboration format are more than welcome. We need to accommodate the different timezones (I’m in CET), but it would be nice to kick off with a (well-prepared) live chat.
2021-06-25T14:31:32Z
Awesome! I’d like to contribute on this project. I have experience fine-tuning dialog models.
2021-06-25T19:59:09Z
It’s a great and very cool idea, i have experience with vision and nlp both, i have read the paper of “Taming Transformers for High-Resolution Image Synthesis” and used the code also,so i think i could positively contribute in this project.
2021-06-26T04:34:51Z
Sounds very interesting. I have experience with generative models and fine-tuning LMs, hope I can contribute in some way!
2021-06-27T21:41:13Z
Hey everyone, just to get things started I’ve created this github repository with the comments from the posts:gif-generation/README.md at main · braadbaart/gif-generation · GitHub. I think I found everyone’s github profiles except bharat and osanseviero -  send me a message on github if you want access.We can use the github repository to discuss issues and datasets (through the issues feature), and add implementations. Just add your code / ideas / comments to the repo and we’ll see who ends up doing what. This is of course not the most efficient way to go about, so let’s evaluate if we can settle on a design and split up the work in a more sensible way in the next couple of days…I have the next four weeks off so I hope to be able to spend some time on thisI work mostly on NLP, and CV is quite new to me. Lots of experience with TF/Keras and Pytorch, not so much with JAX.
2021-06-28T11:14:44Z
Hey guys:)Here’a JAX implem of VQGANGitHub - patil-suraj/vqgan-jax: JAX implementation of VQGANFeel free to use/modify it for this project.
2021-06-28T16:28:34Z
let’s officially define this projectPutting everybody in the official sheethere. More people can still join!  Leave a comment here or on the sheet if you want to change something. If there are more than 10 people we will split it into two groups, as managing more than 10 people will be a bit difficult.
2021-06-29T04:52:09Z
Hey everyone!This seems very exciting. I would love to join in and contribute.I have prior experience in NLP and also been working on some image reconstruction in CV recently.
2021-06-30T04:47:10Z
On discord I have created a channel named#clip-replyplease join there & we can start discussing~
2021-06-30T07:35:48Z
This is one sounds like a super interesting project!
No date available
No reply text available