How to create dataset from CSV to training Question answering？
I made a CSV by referring to the SQUAD data structure. Then trying use script from here to training Question answering using customized CSV.github.comtransformers/examples/pytorch/question-answering at main ·...main/examples/pytorch/question-answering🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX. - transformers/examples/pytorch/question-answering at main · huggingface/transformersWhen converting from CSV to Dataset, I understood that it was necessary to convert the feature in the answers column to Sequence (feature…, so I converted it as follows.#Custom features
ans_feature = Features({'text': Value(dtype='string', id=None), 'answer_start': Value(dtype='int32', id=None)})
custom_features = Features(
    {
        "id": Value(dtype='string', id=None),
        "title": Value(dtype='string', id=None),
        "context": Value(dtype='string', id=None),
        "question": Value(dtype='string', id=None),
        "answers": Sequence(ans_feature, length=-1, id=None),
    }
)

#Create train datasets
df = pd.read_csv("train.csv", encoding="utf_8")
train_l = []
for index, row in df.iterrows():
    train_l.append({'id': row[0], 'title': row[1], 'context': row[2], 'question': row[3], 'answers': row[4]})

train_dataset = Dataset.from_pandas(pd.DataFrame(data=train_l), features=custom_features)

#Create validation datasets
df = pd.read_csv("validation.csv", encoding="utf_8")
train_v = []
for index, row in df.iterrows():
    train_v.append({'id': row[0], 'title': row[1], 'context': row[2], 'question': row[3], 'answers': row[4]})

validation_dataset = Dataset.from_pandas(pd.DataFrame(data=train_v), features=custom_features)

#Create datasets
raw_datasets = DatasetDict({
    "train": train_dataset,
    "validation": validation_dataset,
})However, the conversion process did not go well and the following error occurred.File "/Users/tphan/.pyenv/versions/anaconda3-2023.03/envs/transformers/lib/python3.10/site-packages/datasets/table.py", line 2140, in cast_array_to_feature
    raise TypeError(f"Couldn't cast array of type\n{array.type}\nto\n{feature}")
TypeError: Couldn't cast array of type
string
to
Sequence(feature={'text': Value(dtype='string', id=None), 'answer_start': Value(dtype='int32', id=None)}, length=-1, id=None)Can someone please tell me the correct way and the code?
2023-07-06T13:11:19Z
Can you paste the first few lines of your CSV? What the structure of theanswerscolumn in your CSV?You can use theconvertersparam inread_csvto parse complex columns.Also, JSON (Lines) is much better for representing such data (load_datasetwould work out of the box).
2023-07-07T04:28:37Z
@mariosaskoThank you for replied.I saved the SQUAD data to CSV for testing. Then I tried two patterns of CSV as shown below, but neither could be converted successfully. I haven’t used the converers parameter yet, so I’ll try it.squad_dataset1920×244 86.8 KB
2023-07-07T13:16:04Z
You should be able to parse the second pattern with:pd.read_csv(..., converters={"answers": lambda x: json.loads(x)})
2023-07-10T06:28:25Z
Thank you for your reply. I tried it, but it still doesn’t work with “json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes” error.The data structure doesn’t seem correct, so I’ll try again. Thanks again.
2023-07-13T00:39:00Z
Sharing information for those in need like me. I was finally able to train with the following data structure. I would be happy if you could refer to it. Thanks to@mariosaskovery much again.スクリーンショット 2023-07-13 9.33.531920×337 121 KB