Understanding DataCollation
[Cross-posting fromPytorch forum]I’m interested in theCaseholder dataset, amultiple choice task ala SWAGwhere examplesinclude five separate possible answers. I use a preprocess function like this:def preprocess_function(examples,mapped=True):
        
    # Repeat  first sentence NMultChoice times to go with the NMultChoice possibilities of second sentences.
    first_sentences = [[context] * NMultChoice for context in examples[Context_name]]
    
    # Grab all second sentences possible for each context.
    second_sentences = [ [ f"{examples[str(i+1)][0]}" for i in range(NMultChoice) ] for context in examples[Context_name]]
    
    # Flatten out
    first_sentences = sum(first_sentences, [])
    second_sentences = sum(second_sentences, [])
    
    # Tokenizer will return a list of lists of lists for each key: 
    # a list of all examples (here 5), then a list of all choices (4) and a list of input IDs 
    # (length varying here since we did not apply any padding)
    tokenized_examples = tokenizer(first_sentences, second_sentences, truncation=True)
    
    # Un-flatten so that each example has NMultChoice input ids, attentions masks, etc.
    tokDict = {k: [v[i : i + NMultChoice] for i in range(0, len(v), NMultChoice)] for k, v in tokenized_examples.items()}
        
    tokDict['label'] = examples[Label_name]
    tokDict['idx'] = examples[Idx_name]
    # capture casehold scores
    tokDict['scores'] = {f'{i}': example[f'{i+6}'] for i in range(NMultChoice)}
        
    return tokDictand generate a processed dataset like this:ds = datasets.load_dataset(csvDir,data_files=dsFiles, download_mode="force_redownload")
ds2 = ds.map(preprocessOne)The tokenized dataset has features that looks like this:Dataset({
	features: ['Unnamed: 0', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', 'input_ids', 'token_type_ids', 'attention_mask', 'label', 'idx', 'scores'],
	num_rows: 5314
	})Eg, focusing on theinput_idsgenerated by the tokenizer, it is a list of lists of varying lengths:iid = eval_dataset['input_ids']
	iid00 = iid[0][0]
	for l in iid00: print (len(l))

	> 224
	> 225
	> 229
	> 228
	> 224Also because of the multiple choice task, I’m using a DataCollator modeled afterthis swag example:class DataCollatorForMultipleChoice:

tokenizer: PreTrainedTokenizerBase
padding: Union[bool, str, PaddingStrategy] = True
max_length: Optional[int] = None
pad_to_multiple_of: Optional[int] = None

def __call__(self, features):
    label_name = "label" if "label" in features[0].keys() else Label_name # "labels"
    labels = [feature.pop(label_name) for feature in features]
    batch_size = len(features)
    num_choices = len(features[0]["input_ids"])
    flattened_features = []
    for feature in features:
        mclist = []
        for i in range(NMultChoice):
            mcdict = {}
            mcdict['input_ids'] = feature['input_ids']
            mcdict['token_type_ids'] = feature['token_type_ids']
            mcdict['attention_mask'] = feature['attention_mask']
            mclist.append(mcdict)
        flattened_features.append(mclist)
        
    flattened_features = list(chain(*flattened_features))
        
    batch = self.tokenizer.pad(
        flattened_features,
        padding=self.padding,
        max_length=self.max_length,
        pad_to_multiple_of=self.pad_to_multiple_of,
        return_tensors="pt",
    )
    
    # Un-flatten
    batch = {k: v.view(batch_size, num_choices, -1) for k, v in batch.items()}
    # Add back labels
    batch["labels"] = torch.tensor(labels, dtype=torch.int64)
    return batchThis DataCollator is then used in specification of the Trainer, specificallyspecifying pad_to_multiple_of=8and then using it to create batches:data_collator = DataCollatorForMultipleChoice(tokenizer=tokenizer, pad_to_multiple_of=8),

    eval_loader = trainer.get_eval_dataloader(eval_dataset)
    for batch in eval_loader:Now I am getting this exception:File "/Users/rik/Code/eclipse/ai4law/src/casehold_demo-v2.py", line 444, in main
	for batch in eval_loader:
	File ".../lib/python3.11/site-packages/accelerate/data_loader.py", line 452, in __iter__
	current_batch = next(dataloader_iter)
	^^^^^^^^^^^^^^^^^^^^^
	File ".../lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 630, in __next__
	data = self._next_data()
	^^^^^^^^^^^^^^^^^
	File ".../lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 674, in _next_data
	data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File ".../lib/python3.11/site-packages/torch/utils/data/_utils/fetch.py", line 57, in fetch
	return self.collate_fn(data)
	^^^^^^^^^^^^^^^^^^^^^
	File "/Users/rik/Code/eclipse/ai4law/src/casehold_demo-v2.py", line 157, in __call__
	batch = self.tokenizer.pad(
	^^^^^^^^^^^^^^^^^^^
	File ".../lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 3380, in pad
	return BatchEncoding(batch_outputs, tensor_type=return_tensors)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File ".../lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 224, in __init__
	self.convert_to_tensors(tensor_type=tensor_type, prepend_batch_axis=prepend_batch_axis)
	File ".../lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 775, in convert_to_tensors
	raise ValueError(
	ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your features (`input_ids` in this case) have excessive nesting (inputs type `list` where type `int` is expected).I’m doing truncation as part of tokenization, and padding the batch examples withpad_to_multiple_of=8. Would you knopw why would I be getting this error?