CLIP+NeRF: Fewshot Learning, Putting NeRF on a Diet
ViT+CLIP+NeRF: Fewshot Learning, Putting NeRF on a DietIs anyone interested in Computer Vision Field?Our team project goal is the implementation of this paper:Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis:arXiv.orgPutting NeRF on a Diet: Semantically Consistent Few-Shot View SynthesisWe present DietNeRF, a 3D neural scene representation estimated from a few
images. Neural Radiance Fields (NeRF) learn a continuous volumetric
representation of a scene through multi-view consistency, and can be rendered
from novel viewpoints by ray...With this project, We will code a3D neural scene representation (NeRF: Neural Radiances Field) estimated from a few images. Which is based on extracting the semantic information usinga pre-trained visual encoder such as CLIP, a Vision Transformer.1. Languages and SkillsLanguages : Python, Pytorch, JAXOther Skills : Git, Github, Cloud Computing Experience2. Model / Baseline CodeViT, CLIP :https://github.com/openai/CLIPMeatlearning-NeRF (Learned Initialization Paper : JAX Based NeRF Code) :https://github.com/tancik/learnitNeRF (Pytorch Based NeRF Code) :https://github.com/yenchenlin/nerf-pytorch3. DatasetsPhototourism dataset:https://www.cs.ubc.ca/~kmyi/imw2020/data.htmlRealistic Synthetic dataset :https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1DTU multi-view stereo (MVS) dataset :http://roboimagedata.compute.dtu.dk/?page_id=364. ChallengesJAX based NeRF CodeData PreprocessingCLIP / NeRF training with TPUAugmentation CLIP to NeRF with JAX based code5. Project Goal OutputThe desired project output is an implementation of the Fewshot learning NeRF model which is using CLIP+ViT (JAX based code)6. ReferencesNeRF :https://www.matthewtancik.com/nerfPutting NeRF in a diet :https://arxiv.org/abs/2104.00677Learned Initialization :https://arxiv.org/abs/2012.021897. Team Mindset-Enjoy!-Passionate-Collaborative-Learning Oriented
2021-06-29T15:22:16Z
@SasikanthThank you for your interestWelcome!
2021-06-29T15:27:30Z
Really like the atmosphere hereLooks like one of the more ambitious projects - let’s give it a try!@valhalla- maybe you can take a look here as well. Will put it down on the Excel
2021-06-29T17:58:38Z
I’m interested!
2021-06-29T22:08:05Z
Welcome guys!Let’s make a wonderful project! Please come and join the “putting-nerf-on-a-diet” channel on the discord :Flax-HuggingFace-Community-Week@Sasikanth@JYChung@alexlau
2021-06-30T03:51:49Z
This is really interesting, and would love to be part of this.
2021-06-30T04:07:22Z
I’m interested too!
2021-06-30T07:25:43Z
This looks awesome, I’m in
2021-06-30T07:32:08Z
Nice to meet you all! Our team finishes the team recruiting now!Welcome 8 members:@howtowhy@Sasikanth@JYChung@alexlau@syedmech47@sseung0703@MrBananaHuman@khalidsaifullaah@valhalla- Hello, valhalla, Could you put our team members to exel sheet?
2021-06-30T07:55:36Z
Sure!This looks like an ambitious project!Out of curiosity, how long do you think it’ll take to NeRF in JAX?VITandCLIPare already available in JAX.It’s important to define the scope such that the project can be finished with the given time and compute budget.
2021-06-30T08:13:24Z
@valhallaThank you for fast response.Could I reply after checking more detail on the paper? At now I guess that CLIP and NeRF could be available as JAX code but connecting CLIP and NeRF + Training unified model to new data would be challenge for our team. Also we will consider detail implementation on the paper.
2021-06-30T09:00:56Z
Could I reply after checking more detail on the paperSure!
2021-06-30T14:14:40Z
Is there still space left for one more?I have been looking at Few-shot learning for the past month and experimenting with Resnet architectures, would love to try ViT and CLIP.
2021-07-01T04:53:41Z
@hassiahkOK hassiahk! If you want to join us please come to discord!#putting_nerf_on_a_dietchannelFlax-HuggingFace-Community-WeekThe team recruiting is really finished! now :0!
2021-07-01T10:07:14Z
added you@hassiahk
2021-07-02T05:36:50Z
Hi,I think I check this too late though… May I join?I am a vision researcher in AR industry.I’m interested in this project!!
2021-07-03T01:14:47Z
@george31Hello could you come in to discord of our channel?
2021-07-03T06:52:27Z
@george31If you want to join our team, please come in discord and participate today meeting
2021-07-03T07:04:56Z
howtowhy:Hello could you come in to discord of our channel?Yes!! Thanks!
2021-07-03T07:06:35Z
@howtowhyCan I get a new invitation link? … I thinks the link above got expired…
No date available
No reply text available