ValueError: too many values to unpack (expected 2) when using BertTokenizer
Hi everyone,I get an error when usingBertTokenizer.I doencoding = tokenizer([[prompt, prompt, prompt], [choice0, choice1, choice2]], return_tensors='tf', padding=True))and getValueError: too many values to unpack (expected 2).When I doencoding = tokenizer([[prompt, prompt], [choice0, choice1]], return_tensors='tf', padding=True)it works. Any idea why? I want to fine-tuneTFBertForMultipleChoicesuch that each question (prompt) has three choices and not two as in the documentationBERT — transformers 4.7.0 documentationBelow is the complete codeimport os
import numpy as np
import pandas as pd
import tensorflow as tf

from transformers import BertTokenizer, TFBertForMultipleChoice


tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = TFBertForMultipleChoice.from_pretrained('bert-base-uncased')

prompt = "Accept and check containers of mail from large volume mailers, couriers, and contractors."
choice0 = "Time Management"
choice1 = "Writing"
choice2 = "Reading Comprehension"

encoding = tokenizer([[prompt, prompt, prompt], [choice0, choice1, choice2]], return_tensors='tf', padding=True)
inputs = {k: tf.expand_dims(v, 0) for k, v in encoding.items()}
outputs = model(inputs)  # batch size is 1

logits = outputs.logitsThanks!Ayala
2021-07-13T11:46:44Z
Oh the example has one pairs of [] that is not necessary, will fix. It should be:encoding = tokenizer([prompt, prompt, prompt], [choice0, choice1, choice2], return_tensors='tf', padding=True)
2021-07-13T14:03:24Z
Thanks!But when I doencoding = tokenizer([prompt, prompt, prompt], [choice0, choice1, choice2], return_tensors='tf', padding=True)Theencodinglooks like the following:{'input_ids': <tf.Tensor: shape=(3, 23), dtype=int32, numpy=
array([[  101,  5138,  1998,  4638, 16143,  1997,  5653,  2013,  2312,
         3872,  5653,  2545,  1010, 18092,  2015,  1010,  1998, 16728,
         1012,   102,  2051,  2968,   102],
       [  101,  5138,  1998,  4638, 16143,  1997,  5653,  2013,  2312,
         3872,  5653,  2545,  1010, 18092,  2015,  1010,  1998, 16728,
         1012,   102,  3015,   102,     0],
       [  101,  5138,  1998,  4638, 16143,  1997,  5653,  2013,  2312,
         3872,  5653,  2545,  1010, 18092,  2015,  1010,  1998, 16728,
         1012,   102,  3752, 26683,   102]], dtype=int32)>, 'token_type_ids': <tf.Tensor: shape=(3, 23), dtype=int32, numpy=
array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1,
        1],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1,
        0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1,
        1]], dtype=int32)>, 'attention_mask': <tf.Tensor: shape=(3, 23), dtype=int32, numpy=
array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
        1],
       [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
        0],
       [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
        1]], dtype=int32)>}which as far as I understand is encoded as 3 pairs of texts and not as one question with 3 choices.  Namely, wouldn’t I want the encoding to look something like[101, 5138, ..., 102, 2051, 2968..., 102, 3015, ..., 102, 3752..., 102]In other words, if I want to fine-tuneTFBertForMultipleChoice, don’t I need to encode the prompt and choices asprompt choice0 choice1 choice2?Thanks,Ayala
2021-07-13T14:05:45Z
No, you need tensors of shape1, num_choices, seq_length, which is what the code sample will give you with the second line that does the expand_dims.
2021-07-13T14:41:06Z
Thank you very much, I was about to write the same!I have one more question, hope that is ok. How can I tokenize more than one prompt it’s choices? Namely, I have a batch of prompts and each one has it’s three choices. Cannot figure how to tokenize it. Should I useencoding = tokenizer([prompt, prompt, prompt], [choice0, choice1, choice2], return_tensors='tf', padding=True)
inputs = {k: tf.expand_dims(v, 0) for k, v in encoding.items()}for each prompt and it’s choices one at a time? Or there is a way to tokenize them in one batch?Thanks again,Ayala
2021-07-13T14:47:49Z
You should look at themultiple choice exmaples(PyTorch only but the processing is the same) or thenotebookthat goes with it.
2021-07-13T15:04:00Z
Great! Exactly what I was looking for.Ayala