Can't locate the error in my dataset
I always get the following error when I try to download (or preview) the datasetarnastofnun/test.Traceback (most recent call last):File “/home/starkadur/.local/lib/python3.8/site-packages/datasets/builder.py”, line 1869, in _prepare_split_singlewriter.write_table(table)File “/home/starkadur/.local/lib/python3.8/site-packages/datasets/arrow_writer.py”, line 580, in write_tablepa_table = table_cast(pa_table, self._schema)File “/home/starkadur/.local/lib/python3.8/site-packages/datasets/table.py”, line 2292, in table_castreturn cast_table_to_schema(table, schema)File “/home/starkadur/.local/lib/python3.8/site-packages/datasets/table.py”, line 2245, in cast_table_to_schemaarrays = [File “/home/starkadur/.local/lib/python3.8/site-packages/datasets/table.py”, line 2246, incast_array_to_feature(File “/home/starkadur/.local/lib/python3.8/site-packages/datasets/table.py”, line 1795, in wrapperreturn pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])File “/home/starkadur/.local/lib/python3.8/site-packages/datasets/table.py”, line 1795, inreturn pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])File “/home/starkadur/.local/lib/python3.8/site-packages/datasets/table.py”, line 2005, in cast_array_to_featurearrays = [File “/home/starkadur/.local/lib/python3.8/site-packages/datasets/table.py”, line 2006, in_c(array.field(name) if name in array_fields else null_array, subfeature)File “/home/starkadur/.local/lib/python3.8/site-packages/datasets/table.py”, line 1797, in wrapperreturn func(array, *args, **kwargs)File “/home/starkadur/.local/lib/python3.8/site-packages/datasets/table.py”, line 2102, in cast_array_to_featurereturn array_cast(File “/home/starkadur/.local/lib/python3.8/site-packages/datasets/table.py”, line 1797, in wrapperreturn func(array, *args, **kwargs)File “/home/starkadur/.local/lib/python3.8/site-packages/datasets/table.py”, line 1948, in array_castraise TypeError(f"Couldn’t cast array of type {_short_str(array.type)} to {_short_str(pa_type)}")TypeError: Couldn’t cast array of type string to nullI have tried to split the dataset up into smaller bits to find out where in the data the problem lies, but if I only upload the first half of the dataset, I don’t get any error, and if I only upload the second half of the data set, I don’t get any error either. So I can’t know where the problem lies in the data.  All help is appreciated.
2024-09-26T15:12:51Z
It appears to be a complicated issue that is probably unresolved. A workaround seems to be to pass the dataset as JSON.github.com/huggingface/datasetsDatasets load error for saved github issuesopened05:29PM - 14 Jan 23 UTCfolterj### Describe the bug

Loading a previously downloaded & saved dataset as descr…ibed in the HuggingFace course:

issues_dataset = load_dataset("json", data_files="issues/datasets-issues.jsonl", split="train")

Gives this error:
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset

A work-around I found was to use streaming.

### Steps to reproduce the bug

Reproduce by executing the code provided:
https://huggingface.co/course/chapter5/5?fw=pt
From the heading:
'let’s create a function that can download all the issues from a GitHub repository'

### Expected behavior

No error

### Environment info

Datasets version 2.8.0. Note that version 2.6.1 gives the same error (related to null timestamp).

**[EDIT]**

This is the complete error trace confirming the issue is related to the timestamp (`Couldn't cast array of type timestamp[s] to null`)

```
Using custom data configuration default-950028611d2860c8
Downloading and preparing dataset json/default to [...]/.cache/huggingface/datasets/json/default-950028611d2860c8/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51...
Downloading data files: 100%|██████████| 1/1 [00:00<?, ?it/s]
Extracting data files: 100%|██████████| 1/1 [00:00<00:00, 500.63it/s]
Generating train split: 2619 examples [00:00, 7155.72 examples/s]Traceback (most recent call last):
  File "[...]\miniconda3\envs\HuggingFace\lib\site-packages\datasets\builder.py", line 1831, in _prepare_split_single
    writer.write_table(table)
  File "[...]\miniconda3\envs\HuggingFace\lib\site-packages\datasets\arrow_writer.py", line 567, in write_table
    pa_table = table_cast(pa_table, self._schema)
  File "[...]\miniconda3\envs\HuggingFace\lib\site-packages\datasets\table.py", line 2282, in table_cast
    return cast_table_to_schema(table, schema)
  File "[...]\miniconda3\envs\HuggingFace\lib\site-packages\datasets\table.py", line 2241, in cast_table_to_schema
    arrays = [cast_array_to_feature(table[name], feature) for name, feature in features.items()]
  File "[...]\miniconda3\envs\HuggingFace\lib\site-packages\datasets\table.py", line 2241, in <listcomp>
    arrays = [cast_array_to_feature(table[name], feature) for name, feature in features.items()]
  File "[...]\miniconda3\envs\HuggingFace\lib\site-packages\datasets\table.py", line 1807, in wrapper
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
  File "[...]\miniconda3\envs\HuggingFace\lib\site-packages\datasets\table.py", line 1807, in <listcomp>
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
  File "[...]\miniconda3\envs\HuggingFace\lib\site-packages\datasets\table.py", line 2035, in cast_array_to_feature
    arrays = [_c(array.field(name), subfeature) for name, subfeature in feature.items()]
  File "[...]\miniconda3\envs\HuggingFace\lib\site-packages\datasets\table.py", line 2035, in <listcomp>
    arrays = [_c(array.field(name), subfeature) for name, subfeature in feature.items()]
  File "[...]\miniconda3\envs\HuggingFace\lib\site-packages\datasets\table.py", line 1809, in wrapper
    return func(array, *args, **kwargs)
  File "[...]\miniconda3\envs\HuggingFace\lib\site-packages\datasets\table.py", line 2101, in cast_array_to_feature
    return array_cast(array, feature(), allow_number_to_str=allow_number_to_str)
  File "[...]\miniconda3\envs\HuggingFace\lib\site-packages\datasets\table.py", line 1809, in wrapper
    return func(array, *args, **kwargs)
  File "[...]\miniconda3\envs\HuggingFace\lib\site-packages\datasets\table.py", line 1990, in array_cast
    raise TypeError(f"Couldn't cast array of type {array.type} to {pa_type}")
TypeError: Couldn't cast array of type timestamp[s] to null
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm 2022.1.3\plugins\python\helpers\pydev\pydevconsole.py", line 364, in runcode
    coro = func()
  File "<input>", line 1, in <module>
  File "C:\Program Files\JetBrains\PyCharm 2022.1.3\plugins\python\helpers\pydev\_pydev_bundle\pydev_umd.py", line 198, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "C:\Program Files\JetBrains\PyCharm 2022.1.3\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "[...]\PycharmProjects\TransformersTesting\dataset_issues.py", line 20, in <module>
    issues_dataset = load_dataset("json", data_files="issues/datasets-issues.jsonl", split="train")
  File "[...]\miniconda3\envs\HuggingFace\lib\site-packages\datasets\load.py", line 1757, in load_dataset
    builder_instance.download_and_prepare(
  File "[...]\miniconda3\envs\HuggingFace\lib\site-packages\datasets\builder.py", line 860, in download_and_prepare
    self._download_and_prepare(
  File "[...]\miniconda3\envs\HuggingFace\lib\site-packages\datasets\builder.py", line 953, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "[...]\miniconda3\envs\HuggingFace\lib\site-packages\datasets\builder.py", line 1706, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "[...]\miniconda3\envs\HuggingFace\lib\site-packages\datasets\builder.py", line 1849, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset
Generating train split: 2619 examples [00:19, 7155.72 examples/s]
```github.com/THUDM/LongBench报错TypeError: Couldn't cast array of type list<item: string> to nullopened08:33AM - 26 Feb 24 UTCxxcoco763服务器没办法链接huggingface，只是将pred.py中THU/Longbench的路径换成了本地的/home/eval/LongBench/data,c…onfig文件中的模型路径也已经添加，报错如下
CUDA_VISIBLE_DEVICES=7 python pred.py --model llama2-13b-chat-16k
Resolving data files: 100%|████████████████████████████████████| 34/34 [00:00<00:00, 149169.81it/s]
Downloading data files: 100%|██████████████████████████████████████| 1/1 [00:00<00:00, 1417.95it/s]
Extracting data files: 100%|█████████████████████████████████████████| 1/1 [00:00<00:00, 87.24it/s]
Generating train split: 2500 examples [00:00, 4816.93 examples/s]
Traceback (most recent call last):
  File "/root/miniconda3/envs/zyx/lib/python3.10/site-packages/datasets/builder.py", line 1940, in _prepare_split_single
    writer.write_table(table)
  File "/root/miniconda3/envs/zyx/lib/python3.10/site-packages/datasets/arrow_writer.py", line 572, in write_table
    pa_table = table_cast(pa_table, self._schema)
  File "/root/miniconda3/envs/zyx/lib/python3.10/site-packages/datasets/table.py", line 2328, in table_cast
    return cast_table_to_schema(table, schema)
  File "/root/miniconda3/envs/zyx/lib/python3.10/site-packages/datasets/table.py", line 2287, in cast_table_to_schema
    arrays = [cast_array_to_feature(table[name], feature) for name, feature in features.items()]
  File "/root/miniconda3/envs/zyx/lib/python3.10/site-packages/datasets/table.py", line 2287, in <listcomp>
    arrays = [cast_array_to_feature(table[name], feature) for name, feature in features.items()]
  File "/root/miniconda3/envs/zyx/lib/python3.10/site-packages/datasets/table.py", line 1831, in wrapper
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
  File "/root/miniconda3/envs/zyx/lib/python3.10/site-packages/datasets/table.py", line 1831, in <listcomp>
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
  File "/root/miniconda3/envs/zyx/lib/python3.10/site-packages/datasets/table.py", line 2143, in cast_array_to_feature
    return array_cast(array, feature(), allow_number_to_str=allow_number_to_str)
  File "/root/miniconda3/envs/zyx/lib/python3.10/site-packages/datasets/table.py", line 1833, in wrapper
    return func(array, *args, **kwargs)
  File "/root/miniconda3/envs/zyx/lib/python3.10/site-packages/datasets/table.py", line 2028, in array_cast
    raise TypeError(f"Couldn't cast array of type\n{array.type}\nto\n{pa_type}")
TypeError: Couldn't cast array of type
list<item: string>
to
null

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/zyx/eval/LongBench/pred.py", line 163, in <module>
    data = load_dataset('/root/zyx/eval/LongBench/data/data', dataset, split='test')
  File "/root/miniconda3/envs/zyx/lib/python3.10/site-packages/datasets/load.py", line 2153, in load_dataset
    builder_instance.download_and_prepare(
  File "/root/miniconda3/envs/zyx/lib/python3.10/site-packages/datasets/builder.py", line 954, in download_and_prepare
    self._download_and_prepare(
  File "/root/miniconda3/envs/zyx/lib/python3.10/site-packages/datasets/builder.py", line 1049, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/root/miniconda3/envs/zyx/lib/python3.10/site-packages/datasets/builder.py", line 1813, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "/root/miniconda3/envs/zyx/lib/python3.10/site-packages/datasets/builder.py", line 1958, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset

请问如何解决呢
2024-09-27T06:41:34Z
Hi,This is because the author has just uploaded a zip file to the dataset repo:arnastofnun/test at main, hence one would need to unzip it locally. Here’s how to download the file:from huggingface_hub import REDACTED_TOKEN_download

filepath = REDACTED_TOKEN_download(repo_id="arnastofnun/test", filename="igc_news1_eidfaxi.zip", repo_type="dataset")
2024-09-30T07:59:33Z
Thank your for your answer.I have tried to upload the file unzipped. I have deleted the zip-file and uploaded the jsonl-file,  but the error remains. Actually this dataset is a snippet of a much larger dataset (arnastofnun/IGC-2022-1) with tens of much bigger zip-files, and there I only have this problem with three of the files. So it seems that the zip-files are not the problem.