Error when executing pix2struct-widget-captioning-base model
Hi!I’m trying to run thepix2struct-widget-captioning-basemodel. (link)When I am executing it like described on the model card, I get an error: “ValueError: A header text must be provided for VQA models.”I think the model card description is missing the information how to add the bounding box for locating the widget, the description just says it should be the same as forpix2struct-textcaps-base, which I don’t think is the case.Any idea how to add this information?I’m quite new to using HuggingFace models.Thank you!
2023-03-29T14:43:06Z
I found that the code works if a text question is used.But I’m not sure what format to use for the bounding box. Any example for that?My code:from PIL import Imagefrom transformers import Pix2StructForConditionalGeneration, Pix2StructProcessorfile_path = “…\test_images\full_screenshot.png”image = Image.open(file_path).convert(“RGB”)model = Pix2StructForConditionalGeneration.from_pretrained(“google/pix2struct-widget-captioning-base”)processor = Pix2StructProcessor.from_pretrained(“google/pix2struct-widget-captioning-base”)TODO: add a bounding box instead of the questionquestion = “What does the label 15 represent? (1) lava (2) core (3) tunnel (4) ash cloud”inputs = processor(images=image, text=question, return_tensors=“pt”)predictions = model.generate(**inputs)print(processor.decode(predictions[0], skip_special_tokens=True))
2023-05-25T15:40:44Z
DId you figure out how to use the model?Currently I manually draw bounding boxes inside the image using an image editing tool. But I’m getting the same prediction no matter where I draw the bounding box, so I’m not sure if the model is working.Below is my code:import requests
from PIL import Image
from transformers import Pix2StructForConditionalGeneration, Pix2StructProcessor

model = Pix2StructForConditionalGeneration.from_pretrained("google/pix2struct-widget-captioning-base")
processor = Pix2StructProcessor.from_pretrained("google/pix2struct-widget-captioning-base")
image = Image.open("/path/to/image.png")
inputs = processor(images=image, text="", return_tensors="pt")

predictions = model.generate(**inputs)
print(processor.decode(predictions[0], skip_special_tokens=True))
2023-06-07T08:04:03Z
No, I couldn’t figure it out. I was checking the original google pix2struct github repo a bit, but didn’t find an example. Otherwise I’m not sure how to investigate further. Would be happy to see the solution.
2024-02-11T12:15:58Z
molntamas:, text=questionthe model u are using is a captioning model, thus it will generate a caption for an image, so drawing a bounding box should not normally change the caption of the image - therefor u get the same prediction
2024-02-21T07:59:21Z
molntamas:pix2structHi doc2txt,like the name of the model also implies it is a widget captioning model, we need to provide a bounding box, and the part of the image withing the bounding box gets captioned.See the paper about pix2structhttps://arxiv.org/pdf/2210.03347.pdf, page 6:“Widget Captioning (Li et al., 2020b) is an image captioning task where the input is an app screenshot annotated witha single bounding box denoting a widget (e.g. a button ora scroll bar). The caption describes the functionality of thewidget (e.g. find location). VUT (Li et al., 2021b), the current SotA uses a specialized UI encoder combining images,bounding boxes, and view hierarchies. Pix2StructLarge improves the SotA CIDEr from 127.4 to 136.7.”Regards,Tamas
2024-07-10T08:07:10Z
Hi molntamas,I’m currently try ing to run pix2struct-widget-captioning, and I met the same problem:need to describe bbox instead of texts as a header“ValueError: A header text must be provided for VQA models.”I’m wondering whether you have find a way to solve itThank youRegards,Steve