ValueError: Invalid image type. Expected either PIL.Image.Image, numpy.ndarray, torch.Tensor, tf.Tensor or jax.ndarray, but got
I am trying to train the Deplot model using Huggingface Library but facing Value Error.I follow the pix2struct notebook as suggested in Deplot Code.https://github.com/huggingface/notebooks/blob/main/examples/image_captioning_pix2struct.ipynbI have successfully trained pix2struct-base models using the same data format (image and text) without encountering any issues. However, I face this particular problem when training the “Deplot” model.My Dataset format:Dataset({features: [‘image’, ‘text’],num_rows: 14289})The code below displays “True”, meaning the format is correct.print(isinstance(dataset[0][‘image’], Image.Image))Code I used:from torch.utils.data import Dataset, DataLoaderMAX_PATCHES = 1024class ImageCaptioningDataset(Dataset):definit(self, dataset, processor):self.dataset = datasetself.processor = processordef __len__(self):
    return len(self.dataset)

def __getitem__(self, idx):
    item = self.dataset[idx]
    encoding = self.processor(images=item["image"], text="Generate underlying data table of the figure below:", return_tensors="pt", add_special_tokens=True, max_patches=MAX_PATCHES)
    encoding = {k:v.squeeze() for k,v in encoding.items()}
    encoding["text"] = item["text"]
    return encodingfrom transformers import AutoProcessor, Pix2StructForConditionalGenerationprocessor = AutoProcessor.from_pretrained(“google/deplot”)model = Pix2StructForConditionalGeneration.from_pretrained(“google/deplot”)import torchdef collator(batch):new_batch = {“flattened_patches”:, “attention_mask”:}texts = [item[“text”] for item in batch]text_inputs = processor(text=texts, padding="max_length", truncation=True, return_tensors="pt", add_special_tokens=True, max_length=512)

new_batch["labels"] = text_inputs.input_ids

for item in batch:
    new_batch["flattened_patches"].append(item["flattened_patches"])
    new_batch["attention_mask"].append(item["attention_mask"])

new_batch["flattened_patches"] = torch.stack(new_batch["flattened_patches"])
new_batch["attention_mask"] = torch.stack(new_batch["attention_mask"])

return new_batchtrain_dataset = ImageCaptioningDataset(dataset, processor)train_dataloader = DataLoader(train_dataset, shuffle=True, batch_size=2, collate_fn=collator)import torchfrom transformers.optimization import Adafactor, get_cosine_schedule_with_warmupimport osEPOCHS = 5000optimizer = Adafactor(model.parameters(), scale_parameter=False, relative_step=False, lr=0.01, weight_decay=1e-05)scheduler = get_cosine_schedule_with_warmup(optimizer, num_warmup_steps=1000, num_training_steps=40000)device = torch.device(“cuda:2”) if torch.cuda.is_available() else “cpu”model.to(device)model.train()for epoch in range(EPOCHS):print(“Epoch:”, epoch)for idx, batch in enumerate(train_dataloader): 
    labels = batch.pop("labels").to(device)
    flattened_patches = batch.pop("flattened_patches").to(device)
    attention_mask = batch.pop("attention_mask").to(device)

    outputs = model(flattened_patches=flattened_patches,
                    attention_mask=attention_mask,
                    labels=labels)
    
    loss = outputs.loss

    print("Loss:", loss.item())

    loss.backward()

    optimizer.step()
    optimizer.zero_grad()
    scheduler.step()  # Update the learning rate scheduler

    if (epoch + 1) % 20 == 0:
        model.eval()

        predictions = model.generate(flattened_patches=flattened_patches, attention_mask=attention_mask)        
        print("Predictions:", processor.batch_decode(predictions, skip_special_tokens=True))

        model.train()Error I am facing:ValueError Traceback (most recent call last) Cell In[20], line 18 15 for epoch in range(EPOCHS): 16 print(“Epoch:”, epoch) —> 18 for idx, batch in enumerate(train_dataloader): 19 labels = batch.pop(“labels”).to(device) 20 flattened_patches = batch.pop(“flattened_patches”).to(device) File~/anaconda3/envs/deplot_3/lib/python3.9/site-packages/torch/utils/data/dataloader.py:652, in _BaseDataLoaderIter.next(self) 649 if self._sampler_iter is None: 650 # TODO(Bug in dataloader iterator found by mypy · Issue #76750 · pytorch/pytorch · GitHub) 651 self._reset() # type: ignore[call-arg] → 652 data = self._next_data() 653 self._num_yielded += 1 654 if self._dataset_kind == _DatasetKind.Iterable and \ 655 self._IterableDataset_len_called is not None and \ 656 self._num_yielded > self._IterableDataset_len_called: File~/anaconda3/envs/deplot_3/lib/python3.9/site-packages/torch/utils/data/dataloader.py:692, in _SingleProcessDataLoaderIter._next_data(self) 690 def _next_data(self): 691 index = self._next_index() # may raise StopIteration → 692 data = self._dataset_fetcher.fetch(index) # may raise StopIteration 693 if self._pin_memory: 694 data = _utils.pin_memory.pin_memory(data, self._pin_memory_device)…133 “Invalid image type. Expected either PIL.Image.Image, numpy.ndarray, torch.Tensor, tf.Tensor or " 134 f"jax.ndarray, but got {type(images)}.” 135 )ValueError: Invalid image type. Expected either PIL.Image.Image, numpy.ndarray, torch.Tensor, tf.Tensor or jax.ndarray, but got .
2023-07-24T05:34:27Z
Same question here! My guess is that since our new deplot processor aggregates both the bert-tokenizer processor and the pix2struct processor, it requires ‘images=’ parameter as used in thegetitemmethod from the Dataset class but I have no idea what the images should be in the collator function
2023-08-01T13:44:08Z
This pointed me towards a solution.When running from thenotebook, images and text are processed separately, images inImageCaptioningDataset.__call__and text incollator. Thereforeprocessordoes not get an image input when processing the text. However, sinceprocessor.image_processor.is_vqaisTrue, it expects an image.Inprocessor, the input should be treated as text only, but the conditional evaluates toFalsedue tois_vqa, andself.image_processor(images, ...)gets called withNone, causing the error message.You can do a quick fix by settingprocessor.image_processor.is_vqa = Falsebefore iterating the dataloader.
2023-08-02T01:04:15Z
Thank you, It worked
2024-01-03T16:37:44Z
But from my understanding, the deplot pretraining render the input text “generate the underlying data” as a header to the input images. If we pass theprocessor.image_processor.is_vqa = False, the image wouldn’t get preprocessed correctly since therender_headerwill not be called. Therefore, leading to the incorrect result of input process.So, I don’t think this is a proper solution. Still exploring the correct solutions, will post later
2024-01-05T21:42:52Z
The problem is caused when is_vqa is set to true, they use the pix2struct image processor automatically which requires an image input.Normally, when the processor input doesn’t include an image, it automatically switch back to a simple text tokenizer from image processor, however when is_vqa is set to true in pix2struct auto processor, seems this automatically handling doesn’t work, the processor stick to image processor even your input is text only. Therefore, a simple solution is to use the tokenizer from processor by processor.tokenizer when you would like to tokenize the text inputs, which actually is the ground truth text for model’s output:text_inputs = processor.tokenizer(text=texts, padding="max_length", return_tensors="pt", add_special_tokens=True, max_length=20)I made some change to the dataset and collator as following, which works correctly:class ChartParametersDataset(Dataset):
    def __init__(self, data_root) -> None:
        ...
    
    def __len__(self):
        return len(...)
    
    def __getitem__(self, idx):
        your_code_here
        img = Image.open('yourimagehere.jpg').convert("RGB")
        prompt = "Generate underlying data table of the figure below:"
        text = "your ground truth output text"
        
        inputs = {
            "text":txt,
            "prompt":prompt,
            "image":img}
        return inputs

processor = AutoProcessor.from_pretrained("google/deplot")
model = Pix2StructForConditionalGeneration.from_pretrained("google/deplot")

def collator(batch):
    new_batch = {"flattened_patches":[], "attention_mask":[]}
    texts = [item["text"] for item in batch]
    images = [item["image"] for item in batch]
    prompts = [item["prompt"] for item in batch]
  
    text_inputs = processor.tokenizer(text=texts, padding="max_length", return_tensors="pt", add_special_tokens=True, max_length=20)
  
    new_batch["labels"] = text_inputs.input_ids
  
    encoding = processor(images=images, text=prompts,
                         return_tensors="pt", add_special_tokens=True, 
                         max_patches=1024)

    print(encoding)

    new_batch["flattened_patches"] = encoding["flattened_patches"]
    new_batch["attention_mask"] = encoding["attention_mask"]

    return new_batch
2024-01-09T02:18:01Z
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.