How to find the wrong data from debugging mode in train_dataset.map of run_translation.py
Hello.I got error in executing run_translation.py.the debug log is bellow:Running tokenizer on train dataset:  79% 15/19 [00:03<00:00,  4.72ba/s]05/30/2022 06:02:02 - DEBUG - datasets.arrow_writer - Done writing 15000 examples in 4874189 bytes /root/.cache/huggingface/datasets/aaraki___parquet/aaraki–github-issues7-4ed446480480c542/0.0.0/0b6d5799bb726b24ad7fc7be720c170d8e497f575d02d47537de9a5bac074901/tmpimfoz_hr.Traceback (most recent call last):File “run_translation.py”, line 627, inmain()File “run_translation.py”, line 445, in maindesc=“Running tokenizer on train dataset”,File “/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py”, line 2364, in mapdesc=desc,File “/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py”, line 532, in wrapperout: Union[“Dataset”, “DatasetDict”] = func(self, *args, **kwargs)File “/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py”, line 499, in wrapperout: Union[“Dataset”, “DatasetDict”] = func(self, *args, **kwargs)File “/usr/local/lib/python3.7/dist-packages/datasets/fingerprint.py”, line 458, in wrapperout = func(self, *args, **kwargs)File “/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py”, line 2751, in _map_singlewriter.write_batch(batch)File “/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py”, line 506, in write_batchpa_table = pa.Table.from_arrays(arrays, schema=schema)File “pyarrow/table.pxi”, line 1702, in pyarrow.lib.Table.from_arraysFile “pyarrow/table.pxi”, line 1314, in pyarrow.lib.Table.validateFile “pyarrow/error.pxi”, line 99, in pyarrow.lib.check_statuspyarrow.lib.ArrowInvalid: Column 2 named labels expected length 995 but got length 1000the code is bellow:with training_args.main_process_first(desc=“train dataset map pre-processing”):train_dataset = train_dataset.map(preprocess_function,batched=True,num_proc=data_args.preprocessing_num_workers,remove_columns=column_names,load_from_cache_file=not data_args.overwrite_cache,desc=“Running tokenizer on train dataset”,#error)I tryed to open the /root/ directory in colab,but it didn’t work.Colud you teach me how to fix the wrong data in dataset?