Poster2Plot: Generate Movie/T.V show plot from poster
DescriptionOur team is working on building an image captioning model which can generate a movie/t.v show plot from it’s poster.The goal of this project is to create an image captioning model using a transformer encoder model like Vision Transformer (ViT) and a transformer decoder language model like GPT-2Model(s)Any vision based encoder and language model decoder would be a good candidate to train the VisionEncoderDecoderModel for image captioning. We are trying the following models first:Encoder:google/vit-base-patch16-224-in21kDecoder:gpt2DatasetsWe are using publicly available IMDb datasets to train the model.Some examples:IMDb movies extensive dataset | Kaggle48K IMDB Movies With Posters | KaggleChallengesThe main challenge is to create a good dataset of poster and movie plots. Also it will be interesting to see if the model gives good predictions for non-english movies/tv shows.Desired project outcomesWe will create a Streamlit or Gradio app onSpacesthat can predict a movie/t.v show plot from it’s poster.
2021-11-22T01:21:02Z
Let’s give it a try.
2021-11-22T19:10:00Z
@dk-crazydivand I were able to train aVisionEncoderDecoderModelto generate movie/t.v show plot from poster. We usedgoogle/vit-base-patch16-224-in21kencoder andgpt2decoder.We have uploaded the model tomodel hubposter2plot@lewtunLink to the Gradio app onSpacesposter2plotWe are still working on improving the model.
2021-11-23T14:50:45Z
Wow, this is an incredibly cool project and Space that you’ve created - great job! Thank you for taking part in the course event
2021-11-23T14:56:54Z
Awesome work@dsrand team!
2021-11-24T03:38:50Z
Thank you. It was indeed very fun for us to build and super fun to manually test as well. The course content along with some existing code snippets helped a lot. The ease with which one could build and demo projects like these in HF ecosystem cannot be praised enough. The entire ecosystem cuts down the entire idea to deployment time by at least 10x. With the hub, datasets, abstractions in transformer lib and spaces, we were able to do in a day, what would take probably weeks.