DALL-E - mini version
ProjectRecreate a mini-version of DALL-EUseful links:Project ReportDemoGithub RepoOriginal post:It would be cool to recreate something likeDALL-E.I can see some challenges though:difficult paper though the main concepts could be simplifiedavailability/collection of datasetthe model used was GPT-3, not sure if interesting results can be obtained with smaller modelsmay require more time to implementAlso I don’t believe a public version is available so maybe it will be easier to just do it in Pytorch (I have no experience yet with Flax/JAX).
2021-06-25T03:12:41Z
We Could train NLP based models to generate embedding for sentences and then use this embeddings as input embedding for gan model to generate images. This is pretty simple idea just vice versa of image captioning, but I think it can work pretty well given the performance of transformers based models.
2021-06-25T06:08:31Z
Thats a great idea. Can I join?
2021-06-25T12:00:03Z
Hey@borisThat’s a great idea, I’m super excited about this!Regarding the dataset, here are a few that can be used for thisConceptual Caption- ~3.3M image-text pairsConceptual 12M- 12MWIT- ~37MAlso regarding architecture, seems using a simple scaled-up GPT2 as LM can also give good results (seeCogView). By simple, I mean no sparse attention or row-column attentions, etc which is used in the DALL-E I guess. And GPT2 is already available in JAX. And JAX is way faster on TPU than PTAnd if you look at thisdiscussionit seems an even smaller model could give good enough results on the domain-specific dataset.Also as suggested in the discussion above using the VQGAN fromtaming-transformersas the images tokenizer can further reduce the complexity of training such models as the max image token length for these VQGAN models is 256 (way less than DALL-E’s VQVAE, which uses 1024), so overall 256 text tokens and 256 image tokens = 512, should be manageable on a single v3-8.
2021-06-25T15:52:50Z
Those are really great ideas@Vaibhavbrkn@valhalla(and thanks for the datasets)!@raghav66Anybody can join!Also I noticed lucidrains has a pretty awesome repo that should help so we can pull some items from it as well:DALLE-pytorch
2021-06-25T15:55:25Z
Also I noticed lucidrains has a pretty awesome repo that should help so we can pull some items from it as well:DALLE-pytorchYeah, the linked discussion and the VQGAN idea is from the DALLE-pytorch repoAlso, this just droppedGitHub - google-research/xmcgan_image_generation
2021-06-25T16:34:27Z
I would love to try this as well.
2021-06-28T16:38:08Z
Let’s officially define this projectPutting everybody in the official sheethere. More people can still join! Leave a comment here or on the sheet if you want to change something.
2021-06-29T10:44:38Z
Great idea, would love to join!
2021-06-29T11:11:06Z
Also we could try useGPT-J, a 6bn param version of GPT-3. There is aPR for a HFport open atm too
2021-06-29T23:53:20Z
Thank you@Boris, for opening accepting more people to join. I am very interested in joining as well. Thanks.
2021-06-30T08:46:31Z
Added you to the team
2021-06-30T08:50:16Z
I would love to be involved with this!Yeah, we couldGPT-J, but we might need to choose smaller model (maybe 1B ?) due to the time and compute constraint.The HF PR is for PT, adding JAX version will take some time. But it’s somewhat similar to GPT2 which is already available in flax, which we could use.I wrote aJAX version of VQGANwhich can be used as the image tokenizer.
2021-06-30T10:32:46Z
noice! Yeah I was unsure about the speed of GPT-J, at least its already in jax. Curious to see how much a TPUv3-8 can handle!
2021-06-30T15:18:48Z
I created aDiscord channel here.Let’s try to finalize the details. I tried to summarize all the ideas from the discussion:task: image generation from textmodelimage encoder:JAX version of VQGAN(@valhallaI assume it needs to be trained?)text encoder: JAX GPT2model/GAN structureOption 1: build our own structure as suggested by@Vaibhavbrkn(text embeddings → image GAN) + maybe inspiration fromDALLE-pytorchandCogView. Can be difficult (I always have trouble training GAN’s) but a great learning experience too!Option 2: trainXMC-GAN, I imagine this one is actually fully complete and we would just need to set up the dataset and plug everything in correctly.dataset (thanks@valhalla):Conceptual Caption- ~3.3M image-text pairsConceptual 12M- 12MWIT- ~37Mtraining script: TODOoutcome: model that accepts a sentence and generate a realistic image from itAlso feel free to pick a specific area you would want to focus on!On my side I can also help instrument the training with W&B so we can visualize metrics and some sample images generated live.
2021-06-30T19:12:43Z
I would really like to join this project.I am working as a Computer Vision Research Consultant and can contribute to topics that require knowledge in Vision.DALL-E is something that is quite new and I would like to be a part of this project that recreates it.I am a fast learner and a team player.I have experience working with Python (3 years) and PyTorch (1+ year) and version control (2 years)I have experience in public-speaking, and good communication and writing skills. With this, I will be able to contribute significantly towards creating the demo.I am especially interested in this becauseit democratizes Dall-E and lowers the barrier to use this model, as I have special interest in such projects.Please consider my request to join in this effort. Thanks.
2021-06-30T19:24:13Z
I would love to participate in this project as well. Please let me know what I can do to help
2021-06-30T21:03:00Z
@patrickvonplatenwill you pleaseofficiallyadd me?(and also add to the Sheets document)Received confirmation from@borison Discord.
2021-06-30T21:05:53Z
@patrickvonplatenI would also like to be also officially added, thanks!
2021-07-01T08:22:55Z
I would like to join this project as well if there’s still open slot.@valhalla@boris
No date available
No reply text available