Image Captioning - ViT + BERT with WIT
Hi,So am fairly new at NLP, so im just getting used to HF and NLP in general. My project is atimg-caption in spanish. Thus following questions,IT looks likewitdataset isnt on HF library, so downloaded it on the TPU. But files are on tsv and the HF parsers I found support csv, py dicts, json etcs. What should be the fastest way here to parse correctly? any HF library for TSVs?Valhallamy proj lead suggested usingrun_summarization_flax.pyscript, but I am a bit confused. How would I go about specifying ViT as encoder; and how would I go about adding my cross-attn layer to BERT and specifying it to that script ? I see only onemodel_dirandtokenizer_namewhich I am not sure for my use case.I just want to start with pre-trained networsk and once it works, go with fine tuning etc. Thanks
2021-07-07T09:22:18Z
Hi there,Regrading the dataset:Yes, the dataset is not available indatasetslib. You will need to download the tsv and the prepare the dataset by downloading the image. The tsv file for wit contains the image URLs and other metadata.This script mighthelp. It’s for downloading conceptual captions data, but you could re-purpose it to download WIT.Regarding model:There is no off-the-shelf model for this intransformers(yet!). What we need here is a Seq2Seq model. Where that image encoder is a pre-trained vision model likeViTorCLIP's vision model and the decoder is any pre-trained text model (BERT/ROBERTa).To do this we will need to modify the BERT/ROBERTa model and add a cross-attention layer in it.The encoder will the encode images, the decoder will take thedecoder_input_idsand theencoder_hidden_states.This should be similar to encoder-decoder model in pytorchtransformers/modeling_encoder_decoder.py at master · huggingface/transformers · GitHub.@gchhablaniis also working on a similar project and might have the modeling code readyLet me know if you have any more question.
2021-10-21T07:19:24Z
Any progress on this topic?