DeBERTa - ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length
I’m trying to train a number of transformer models on a classification task. My dataset has only two columns: text and label.As part of the pre-processing, I tokenize, pad, and truncate the texts. The input for this function is a datasets.dataset_dict.DatasetDict object, where ‘checkpoint’ refers to the transformer model been trained and ‘dataset_dict’ refers to the pandas dataframe that is been tokenized.The pre-processing is done with the following script.from transformers import AutoTokenizer, DataCollatorWithPadding
from torch.utils.data import DataLoader

def tokenizer_padding(dataset_dict, checkpoint, batch_size):
    tokenizer = AutoTokenizer.from_pretrained(checkpoint)
    data_collator = DataCollatorWithPadding(tokenizer=tokenizer, padding='max_length')

    def tokenize_function(examples):
        return tokenizer(examples["text"], truncation=True)

    tokenized_dataset = dataset_dict.map(tokenize_function, batched=True)
    tokenized_dataset.set_format("torch", columns=["input_ids", "attention_mask", "label"])

    train_dataloader = DataLoader(
        tokenized_dataset["train"], shuffle=True, batch_size=batch_size, collate_fn=data_collator
    )
    validation_dataloader = DataLoader(
        tokenized_dataset["validation"], batch_size=batch_size, collate_fn=data_collator
    )
    test_dataloader = DataLoader(
        tokenized_dataset["test"], batch_size=batch_size, collate_fn=data_collator
    )

    return train_dataloader, validation_dataloader, test_dataloaderWhen I train for BERT and RoBERTa, the tokenizer_padding function works perfetly. However, when I use deberta (microsoft/deberta-v3-base), I get the following error:ValueError: Couldn’t instantiate the backend tokenizer from one of:(1) atokenizerslibrary serialization file,(2) a slow tokenizer instance to convert or(3) an equivalent slow tokenizer class to instantiate and convert.You need to have sentencepiece installed to convert a slow tokenizer to a fast one.With sentencepiece installed, I get this second error:ValueError: Unable to create tensor, you should probably activate truncation and/or padding with ‘padding=True’ ‘truncation=True’ to have batched tensors with the same length. Perhaps your features (input_idsin this case) have excessive nesting (inputs typelistwhere typeintis expected).I have no idea why only the DeBERTa model fails. Can someone help me with that?
2023-10-03T17:43:49Z
Hope your issue is resolved by now. If not, you could use some of these steps:When loading the tokenizer, you could use the use_fast=False parameter to set the slow tokenizer instead of fast one: eg. tokenizer = AutoTokenizer.from_pretrained(DEBERTA_MODEL, use_fast=False)If you want to use the fast tokenizer, install sentencepiece and then restart the kernel for the update to take effect.If the ValueError: Unable to create tensor, arises, you could add padding=‘max_length’ or ‘longest’ instead of True.
2023-10-03T18:24:40Z
Thanks,@Sandy1857. After some head-scratching, the final solution did the trick. It appears that ‘deberta’ processes inputs differently, requiring me to tweak the padding argument. When I set it to padding=‘longest’, it worked fine.