For google/deplot, what should I input as header text for fine-tuning?
I am trying to do fine-tuning google/deplot according to the link and Notebook below.link:DePlotNotebook:https://github.com/huggingface/notebooks/blob/main/examples/image_captioning_pix2struct.ipynbbut, I got error, “ValueError: A header text must be provided for VQA models.” from following code.encoding = self.processor(images=item[“image”], return_tensors=“pt”, add_special_tokens=True, max_patches=MAX_PATCHES)For deplot, what should I input as header text?
2023-04-25T16:50:19Z
cc@ybelkada
2023-05-10T16:03:24Z
Hi@sinchir0Deplot is a VQA model, so you need to render a question or a specific task directly on the image as the snippet here:google/deplot · Hugging FaceThis is different from image captioning task where the input is image only, and you’re trying to predict a caption given that image
2023-05-10T21:27:30Z
@ybelkadaThank you!I would like to fine tuning Deplot for the task of generating the underlying data tables(plot-to-table).In that case, should I always give the text “Generate underlying data table of the figure below:”?Example code:encoding = self.processor(images=image, text="Generate underlying data table of the figure below:", return_tensors="pt")
2023-05-11T09:24:34Z
hi@sinchir0I think that would work yes! To double check I would also ask the authors directly by opening an issue in the Hub repo of deplotI will also ask the authors on my side and let you know!Thanks!
2023-05-11T09:35:28Z
Hey@sinchir0, thanks for looking into deplot! I’m an author.Yes indeed we always rendered the header “Generate underlying data table of the figure below:” during deplot training. Though in this case I imagine sending just an empty string would also work.
2023-05-11T13:18:47Z
@ybelkada@fl399Thank you! My understanding is now very clear. Now I can proceed with the fine-tuning of deplot!To double check I would also ask the authors directly by opening an issue in the Hub repo of deplotYou’re right, I should have asked in the Hub issue.I have posted an additional question on the Hub issue about the format of the input data table for fine-tuning deplot, if you would like to check it out.huggingface.cogoogle/deplot · For fine-tuning deplot, what form of text should be given as...I am trying to do fine-tuning google/deplot according to the link and Notebook below.
2023-05-11T13:50:43Z
Awesome! Thanks@fl399for replying!