Strange bug in loading local JSON files, using load_dataset

This is the actual error:
```
Failed to read file '/home/lakala/hjc/code/pycode/glm/temp.json' with error <class 'pyarrow.lib.ArrowInvalid'>: cannot mix list and non-list, non-null values
```
Which means some samples are incorrectly formatted.

PyArrow, a storage backend that we use under the hood, requires that all the list elements have the same level of nesting (same number of dimensions) or are `None`.
```python
import pyarrow as pa
pa.array([[1, 2, 3], 2]) # ArrowInvalid: cannot mix list and non-list, non-null values
pa.array([[1, 2, 3], [2]]) # works
```

@mariosasko
I used the same operation to check the original data before and after slicing.
This is reflected in my code.
160000 is not a specific number.
I can also get output using 150000.
This doesn't seem to align very well with what you said.
Because if only some sample formats are incorrect.
So there should be an error in one of the front and back slices.
thank you for your reply.

Our JSON loader does the following in your case:

```python
import json
import pyarrow as pa

with open(file, encoding="utf-8") as f:
    dataset = json.load(f)
keys = set().union(*[row.keys() for row in dataset])
mapping = {col: [row.get(col) for row in dataset] for col in keys}
pa_table = pa.Table.from_pydict(mapping) # the ArrowInvalid error comes from here
```

So if this code throws an error with correctly-formatted JSON, then this is an Arrow bug and should be reported in their repo.

> I used the same operation to check the original data before and after slicing.
This is reflected in my code.
160000 is not a specific number.
I can also get output using 150000.
This doesn't seem to align very well with what you said.
Because if only some sample formats are incorrect.
So there should be an error in one of the front and back slices.

You should shuffle the data to make sure that's not the case

@mariosasko
Thank you.
I will try again.

### Describe the bug

I am using 'load_dataset 'loads a JSON file, but I found a strange bug: an error will be reported when the length of the JSON file exceeds 160000 (uncertain exact number). I have checked the data through the following code and there are no issues. So I cannot determine the true reason for this error.

The data is a list containing a dictionary. As follows:

[
{'input': 'someting...', 'target': 'someting...', 'type': 'someting...', 'history': ['someting...', ...]},
...
]

### Steps to reproduce the bug

```
import json
from datasets import load_dataset

path = "target.json"
temp_path = "temp.json"

with open(path, "r") as f:
    data = json.load(f)
    print(f"\n-------the JSON file length is: {len(data)}-------\n")

with open(temp_path, "w") as f:
    json.dump(data[:160000], f)
dataset = load_dataset("json", data_files=temp_path)
print("\n-------This works when the JSON file length is 160000-------\n")

with open(temp_path, "w") as f:
    json.dump(data[160000:], f)
dataset = load_dataset("json", data_files=temp_path)
print("\n-------This works and eliminates data issues-------\n")

with open(temp_path, "w") as f:
    json.dump(data[:170000], f)
dataset = load_dataset("json", data_files=temp_path)
```

### Expected behavior

```
-------the JSON file length is: 173049-------

Downloading and preparing dataset json/default to /root/.cache/huggingface/datasets/json/default-acf3c7f418c5f4b4/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4...
Downloading data files: 100%|███████████████████| 1/1 [00:00<00:00, 3328.81it/s]
Extracting data files: 100%|█████████████████████| 1/1 [00:00<00:00, 639.47it/s]
Dataset json downloaded and prepared to /root/.cache/huggingface/datasets/json/default-acf3c7f418c5f4b4/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4. Subsequent calls will reuse this data.
100%|████████████████████████████████████████████| 1/1 [00:00<00:00, 265.85it/s]

-------This works when the JSON file length is 160000-------

Downloading and preparing dataset json/default to /root/.cache/huggingface/datasets/json/default-a42f04b263ceea6a/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4...
Downloading data files: 100%|███████████████████| 1/1 [00:00<00:00, 2038.05it/s]
Extracting data files: 100%|█████████████████████| 1/1 [00:00<00:00, 794.83it/s]
Dataset json downloaded and prepared to /root/.cache/huggingface/datasets/json/default-a42f04b263ceea6a/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4. Subsequent calls will reuse this data.
100%|████████████████████████████████████████████| 1/1 [00:00<00:00, 681.00it/s]

-------This works and eliminates data issues-------

Downloading and preparing dataset json/default to /root/.cache/huggingface/datasets/json/default-63f391c89599c7b0/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4...
Downloading data files: 100%|███████████████████| 1/1 [00:00<00:00, 3682.44it/s]
Extracting data files: 100%|█████████████████████| 1/1 [00:00<00:00, 788.70it/s]
Generating train split: 0 examples [00:00, ? examples/s]Failed to read file '/home/lakala/hjc/code/pycode/glm/temp.json' with error <class 'pyarrow.lib.ArrowInvalid'>: cannot mix list and non-list, non-null values
Traceback (most recent call last):
  File "/home/lakala/conda/envs/glm/lib/python3.8/site-packages/datasets/builder.py", line 1858, in _prepare_split_single
    for _, table in generator:
  File "/home/lakala/conda/envs/glm/lib/python3.8/site-packages/datasets/packaged_modules/json/json.py", line 146, in _generate_tables
    raise ValueError(f"Not able to read records in the JSON file at {file}.") from None
ValueError: Not able to read records in the JSON file at /home/lakala/hjc/code/pycode/glm/temp.json.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/lakala/hjc/code/pycode/glm/test.py", line 22, in <module>
    dataset = load_dataset("json", data_files=temp_path)
  File "/home/lakala/conda/envs/glm/lib/python3.8/site-packages/datasets/load.py", line 1797, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/lakala/conda/envs/glm/lib/python3.8/site-packages/datasets/builder.py", line 890, in download_and_prepare
    self._download_and_prepare(
  File "/home/lakala/conda/envs/glm/lib/python3.8/site-packages/datasets/builder.py", line 985, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/lakala/conda/envs/glm/lib/python3.8/site-packages/datasets/builder.py", line 1746, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "/home/lakala/conda/envs/glm/lib/python3.8/site-packages/datasets/builder.py", line 1891, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset
```

### Environment info

```
Ubuntu==22.04

python==3.8

pytorch-transformers==1.2.0
transformers== 4.27.1
datasets==2.12.0
numpy==1.24.3
pandas==1.5.3
```


Better filenotfound for gated

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006374 / 0.011353 (-0.004979) | 0.004100 / 0.011008 (-0.006909) | 0.104031 / 0.038508 (0.065523) | 0.035186 / 0.023109 (0.012076) | 0.328904 / 0.275898 (0.053006) | 0.361409 / 0.323480 (0.037929) | 0.003855 / 0.007986 (-0.004130) | 0.004140 / 0.004328 (-0.000189) | 0.080406 / 0.004250 (0.076156) | 0.045658 / 0.037052 (0.008606) | 0.341133 / 0.258489 (0.082644) | 0.372688 / 0.293841 (0.078847) | 0.032025 / 0.128546 (-0.096521) | 0.008877 / 0.075646 (-0.066769) | 0.354784 / 0.419271 (-0.064488) | 0.068874 / 0.043533 (0.025341) | 0.335441 / 0.255139 (0.080302) | 0.356498 / 0.283200 (0.073298) | 0.113367 / 0.141683 (-0.028316) | 1.522458 / 1.452155 (0.070304) | 1.608046 / 1.492716 (0.115329) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.231653 / 0.018006 (0.213647) | 0.446678 / 0.000490 (0.446188) | 0.003246 / 0.000200 (0.003046) | 0.000085 / 0.000054 (0.000030) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025299 / 0.037411 (-0.012112) | 0.111440 / 0.014526 (0.096914) | 0.118758 / 0.176557 (-0.057799) | 0.175037 / 0.737135 (-0.562098) | 0.124583 / 0.296338 (-0.171755) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.418694 / 0.215209 (0.203484) | 4.174695 / 2.077655 (2.097041) | 1.890323 / 1.504120 (0.386203) | 1.683300 / 1.541195 (0.142106) | 1.781954 / 1.468490 (0.313464) | 0.546131 / 4.584777 (-4.038645) | 3.768055 / 3.745712 (0.022343) | 1.839878 / 5.269862 (-3.429983) | 1.111877 / 4.565676 (-3.453800) | 0.068568 / 0.424275 (-0.355707) | 0.011950 / 0.007607 (0.004343) | 0.527469 / 0.226044 (0.301425) | 5.274887 / 2.268929 (3.005958) | 2.391274 / 55.444624 (-53.053351) | 2.063837 / 6.876477 (-4.812640) | 2.140627 / 2.142072 (-0.001445) | 0.681508 / 4.805227 (-4.123719) | 0.148203 / 6.500664 (-6.352461) | 0.064456 / 0.075469 (-0.011013) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.221478 / 1.841788 (-0.620310) | 14.713705 / 8.074308 (6.639397) | 14.674184 / 10.191392 (4.482792) | 0.148411 / 0.680424 (-0.532012) | 0.017858 / 0.534201 (-0.516343) | 0.436166 / 0.579283 (-0.143117) | 0.437290 / 0.434364 (0.002926) | 0.521994 / 0.540337 (-0.018343) | 0.635488 / 1.386936 (-0.751448) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006108 / 0.011353 (-0.005245) | 0.003888 / 0.011008 (-0.007120) | 0.078424 / 0.038508 (0.039916) | 0.033618 / 0.023109 (0.010509) | 0.376284 / 0.275898 (0.100386) | 0.396957 / 0.323480 (0.073477) | 0.003799 / 0.007986 (-0.004187) | 0.003160 / 0.004328 (-0.001168) | 0.078358 / 0.004250 (0.074107) | 0.045597 / 0.037052 (0.008545) | 0.386396 / 0.258489 (0.127907) | 0.412985 / 0.293841 (0.119144) | 0.031610 / 0.128546 (-0.096936) | 0.008720 / 0.075646 (-0.066926) | 0.085944 / 0.419271 (-0.333328) | 0.050780 / 0.043533 (0.007247) | 0.378099 / 0.255139 (0.122960) | 0.381894 / 0.283200 (0.098694) | 0.098926 / 0.141683 (-0.042756) | 1.513842 / 1.452155 (0.061688) | 1.595040 / 1.492716 (0.102323) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.208169 / 0.018006 (0.190163) | 0.431653 / 0.000490 (0.431163) | 0.000935 / 0.000200 (0.000735) | 0.000088 / 0.000054 (0.000034) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029600 / 0.037411 (-0.007812) | 0.116936 / 0.014526 (0.102410) | 0.125603 / 0.176557 (-0.050953) | 0.177007 / 0.737135 (-0.560129) | 0.130602 / 0.296338 (-0.165736) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.457158 / 0.215209 (0.241949) | 4.563254 / 2.077655 (2.485599) | 2.303549 / 1.504120 (0.799429) | 2.107269 / 1.541195 (0.566074) | 2.130861 / 1.468490 (0.662371) | 0.548931 / 4.584777 (-4.035846) | 3.745578 / 3.745712 (-0.000134) | 1.820372 / 5.269862 (-3.449490) | 1.099316 / 4.565676 (-3.466361) | 0.068218 / 0.424275 (-0.356057) | 0.012336 / 0.007607 (0.004728) | 0.569721 / 0.226044 (0.343676) | 5.691312 / 2.268929 (3.422384) | 2.797483 / 55.444624 (-52.647141) | 2.422621 / 6.876477 (-4.453855) | 2.426187 / 2.142072 (0.284115) | 0.674777 / 4.805227 (-4.130451) | 0.144855 / 6.500664 (-6.355809) | 0.065805 / 0.075469 (-0.009664) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.305078 / 1.841788 (-0.536709) | 14.874315 / 8.074308 (6.800007) | 14.541301 / 10.191392 (4.349909) | 0.175818 / 0.680424 (-0.504606) | 0.018169 / 0.534201 (-0.516032) | 0.435836 / 0.579283 (-0.143447) | 0.458397 / 0.434364 (0.024033) | 0.506232 / 0.540337 (-0.034106) | 0.605306 / 1.386936 (-0.781630) |

![](https://cml.dev/watermark.png#7e0c1ceab96821c7c6557482d25a9bd2078d716a "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006138 / 0.011353 (-0.005215) | 0.003792 / 0.011008 (-0.007216) | 0.099417 / 0.038508 (0.060908) | 0.028739 / 0.023109 (0.005630) | 0.302835 / 0.275898 (0.026937) | 0.336397 / 0.323480 (0.012918) | 0.003537 / 0.007986 (-0.004449) | 0.002973 / 0.004328 (-0.001355) | 0.077461 / 0.004250 (0.073211) | 0.039493 / 0.037052 (0.002440) | 0.302367 / 0.258489 (0.043878) | 0.344936 / 0.293841 (0.051095) | 0.027813 / 0.128546 (-0.100733) | 0.008591 / 0.075646 (-0.067055) | 0.318975 / 0.419271 (-0.100297) | 0.045971 / 0.043533 (0.002438) | 0.301672 / 0.255139 (0.046533) | 0.328202 / 0.283200 (0.045003) | 0.091400 / 0.141683 (-0.050282) | 1.487215 / 1.452155 (0.035060) | 1.557730 / 1.492716 (0.065014) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.208343 / 0.018006 (0.190336) | 0.426764 / 0.000490 (0.426275) | 0.001196 / 0.000200 (0.000996) | 0.000069 / 0.000054 (0.000015) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024332 / 0.037411 (-0.013079) | 0.101861 / 0.014526 (0.087335) | 0.108669 / 0.176557 (-0.067888) | 0.172042 / 0.737135 (-0.565093) | 0.113048 / 0.296338 (-0.183290) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.421419 / 0.215209 (0.206210) | 4.200816 / 2.077655 (2.123162) | 1.913516 / 1.504120 (0.409396) | 1.712167 / 1.541195 (0.170972) | 1.762129 / 1.468490 (0.293639) | 0.561616 / 4.584777 (-4.023161) | 3.398122 / 3.745712 (-0.347590) | 1.744323 / 5.269862 (-3.525538) | 1.036023 / 4.565676 (-3.529653) | 0.067658 / 0.424275 (-0.356617) | 0.011145 / 0.007607 (0.003538) | 0.522803 / 0.226044 (0.296759) | 5.226245 / 2.268929 (2.957317) | 2.355148 / 55.444624 (-53.089476) | 2.014939 / 6.876477 (-4.861538) | 2.140028 / 2.142072 (-0.002044) | 0.695049 / 4.805227 (-4.110178) | 0.138428 / 6.500664 (-6.362236) | 0.066721 / 0.075469 (-0.008748) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.219610 / 1.841788 (-0.622177) | 14.239576 / 8.074308 (6.165268) | 14.381955 / 10.191392 (4.190563) | 0.131208 / 0.680424 (-0.549216) | 0.016698 / 0.534201 (-0.517503) | 0.361373 / 0.579283 (-0.217910) | 0.382560 / 0.434364 (-0.051804) | 0.419427 / 0.540337 (-0.120911) | 0.508314 / 1.386936 (-0.878622) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006174 / 0.011353 (-0.005179) | 0.003893 / 0.011008 (-0.007115) | 0.079614 / 0.038508 (0.041106) | 0.028685 / 0.023109 (0.005576) | 0.368627 / 0.275898 (0.092729) | 0.411599 / 0.323480 (0.088119) | 0.003573 / 0.007986 (-0.004413) | 0.002989 / 0.004328 (-0.001340) | 0.078653 / 0.004250 (0.074402) | 0.041146 / 0.037052 (0.004094) | 0.362387 / 0.258489 (0.103898) | 0.417234 / 0.293841 (0.123393) | 0.027958 / 0.128546 (-0.100589) | 0.008695 / 0.075646 (-0.066952) | 0.084637 / 0.419271 (-0.334635) | 0.044188 / 0.043533 (0.000655) | 0.358514 / 0.255139 (0.103375) | 0.392314 / 0.283200 (0.109114) | 0.093986 / 0.141683 (-0.047697) | 1.535366 / 1.452155 (0.083212) | 1.605978 / 1.492716 (0.113262) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.196215 / 0.018006 (0.178209) | 0.429403 / 0.000490 (0.428913) | 0.003736 / 0.000200 (0.003536) | 0.000078 / 0.000054 (0.000023) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025281 / 0.037411 (-0.012130) | 0.104325 / 0.014526 (0.089799) | 0.111548 / 0.176557 (-0.065009) | 0.162326 / 0.737135 (-0.574809) | 0.113853 / 0.296338 (-0.182486) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.447600 / 0.215209 (0.232391) | 4.463422 / 2.077655 (2.385767) | 2.168028 / 1.504120 (0.663908) | 1.968699 / 1.541195 (0.427504) | 2.035531 / 1.468490 (0.567041) | 0.564575 / 4.584777 (-4.020202) | 3.435338 / 3.745712 (-0.310374) | 2.981930 / 5.269862 (-2.287932) | 1.492172 / 4.565676 (-3.073505) | 0.067981 / 0.424275 (-0.356294) | 0.011254 / 0.007607 (0.003647) | 0.544385 / 0.226044 (0.318340) | 5.441694 / 2.268929 (3.172765) | 2.650168 / 55.444624 (-52.794456) | 2.333974 / 6.876477 (-4.542503) | 2.383424 / 2.142072 (0.241351) | 0.669814 / 4.805227 (-4.135414) | 0.135456 / 6.500664 (-6.365209) | 0.067067 / 0.075469 (-0.008402) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.313275 / 1.841788 (-0.528513) | 14.527636 / 8.074308 (6.453328) | 14.470957 / 10.191392 (4.279565) | 0.144361 / 0.680424 (-0.536063) | 0.016847 / 0.534201 (-0.517354) | 0.365158 / 0.579283 (-0.214125) | 0.393809 / 0.434364 (-0.040555) | 0.428527 / 0.540337 (-0.111810) | 0.515816 / 1.386936 (-0.871120) |

![](https://cml.dev/watermark.png#7845d4c3c301226b3f8941ac90aaa123bfd7c69e "CML watermark")

close https://github.com/huggingface/datasets/issues/5953

<img width="1292" alt="image" src="https://github.com/huggingface/datasets/assets/42851186/270fe5bc-1739-4878-b7bc-ab6d35336d4d">


Bad error message when trying to download gated dataset

cc @sanchit-gandhi @Vaibhavs10 @lhoestq - this is mainly for demos that use Common Voice datasets as done here: https://github.com/facebookresearch/fairseq/tree/main/examples/mms#-transformers

Hi ! the error for me is

```
FileNotFoundError: Couldn't find a dataset script at /content/mozilla-foundation/common_voice_13_0/common_voice_13_0.py or any data file in the same directory. Couldn't find 'mozilla-foundation/common_voice_13_0' on the Hugging Face Hub either: FileNotFoundError: Dataset 'mozilla-foundation/common_voice_13_0' doesn't exist on the Hub. If the repo is private or gated, make sure to log in with `huggingface-cli login`.
```

And tbh idk how you managed to get your error. "n_shards.json" is not even a thing in `datasets`

Okay, I am able to reproduce @patrickvonplaten's original error: https://github.com/Vaibhavs10/scratchpad/blob/main/cv13_datasets_test.ipynb

Also not sure why it looks for `n_shards.json`

Ok I see, this file is downloaded from the CV dataset script - let me investigate

Ok I see: when you log out you no longer have access to the repository.

Therefore the dataset script is loaded from cache:
```
WARNING:datasets.load:Using the latest cached version of the module from /root/.cache/huggingface/modules/datasets_modules/datasets/mozilla-foundation--common_voice_13_0/22809012aac1fc9803eaffc44122e4149043748e93933935d5ea19898587e4d7 (last modified on Wed Jun 14 10:13:17 2023) since it couldn't be found locally at mozilla-foundation/common_voice_13_0., or remotely on the Hugging Face Hub.
```

and the script tries to download the n_shards.json but fails

Is this ok for you https://github.com/huggingface/datasets/pull/5954 ?

I'll do a release this afternoon

this is included in the new release 2.13.0

### Describe the bug

When I attempt to download a model from the Hub that is gated without being logged in, I get a nice error message. E.g.:

E.g.
```sh
Repository Not Found for url: https://huggingface.co/api/models/DeepFloyd/IF-I-XL-v1.0.
Please make sure you specified the correct `repo_id` and `repo_type`.
If you are trying to access a private or gated repo, make sure you are authenticated.
Invalid username or password..
Will try to load from local cache.
```

If I do the same for a gated dataset on the Hub, I'm not gated a nice error message IMO:

```sh
File ~/hf/lib/python3.10/site-packages/fsspec/implementations/http.py:430, in HTTPFileSystem._info(self, url, **kwargs)
    427     except Exception as exc:
    428         if policy == "get":
    429             # If get failed, then raise a FileNotFoundError
--> 430             raise FileNotFoundError(url) from exc
    431         logger.debug(str(exc))
    433 return {"name": url, "size": None, **info, "type": "file"}

FileNotFoundError: https://huggingface.co/datasets/mozilla-foundation/common_voice_13_0/resolve/main/n_shards.json
```

### Steps to reproduce the bug

```
huggingface-cli logout
```

and then:

```py
from datasets import load_dataset, Audio

# English
stream_data = load_dataset("mozilla-foundation/common_voice_13_0", "en", split="test", streaming=True)
stream_data = stream_data.cast_column("audio", Audio(sampling_rate=16000))
en_sample = next(iter(stream_data))["audio"]["array"]

# Swahili
stream_data = load_dataset("mozilla-foundation/common_voice_13_0", "sw", split="test", streaming=True)
stream_data = stream_data.cast_column("audio", Audio(sampling_rate=16000))
sw_sample = next(iter(stream_data))["audio"]["array"]
```

### Expected behavior

Better error message

### Environment info

Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 2.12.0
- Platform: Linux-6.2.0-76060200-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.16.0.dev0
- PyArrow version: 11.0.0
- Pandas version: 1.5.3


Add Arrow builder docs

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006522 / 0.011353 (-0.004831) | 0.004319 / 0.011008 (-0.006690) | 0.099280 / 0.038508 (0.060772) | 0.033117 / 0.023109 (0.010007) | 0.339392 / 0.275898 (0.063494) | 0.366219 / 0.323480 (0.042739) | 0.003896 / 0.007986 (-0.004090) | 0.003412 / 0.004328 (-0.000916) | 0.076655 / 0.004250 (0.072404) | 0.045203 / 0.037052 (0.008150) | 0.355800 / 0.258489 (0.097311) | 0.372533 / 0.293841 (0.078692) | 0.032318 / 0.128546 (-0.096229) | 0.009030 / 0.075646 (-0.066616) | 0.328701 / 0.419271 (-0.090571) | 0.052891 / 0.043533 (0.009358) | 0.341131 / 0.255139 (0.085992) | 0.351593 / 0.283200 (0.068393) | 0.105136 / 0.141683 (-0.036546) | 1.475953 / 1.452155 (0.023798) | 1.566074 / 1.492716 (0.073357) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.216671 / 0.018006 (0.198664) | 0.446952 / 0.000490 (0.446462) | 0.006340 / 0.000200 (0.006140) | 0.000096 / 0.000054 (0.000042) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028293 / 0.037411 (-0.009118) | 0.112298 / 0.014526 (0.097773) | 0.118634 / 0.176557 (-0.057923) | 0.175542 / 0.737135 (-0.561593) | 0.124773 / 0.296338 (-0.171565) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.435209 / 0.215209 (0.220000) | 4.344361 / 2.077655 (2.266706) | 2.128943 / 1.504120 (0.624823) | 1.945465 / 1.541195 (0.404271) | 2.049932 / 1.468490 (0.581442) | 0.547126 / 4.584777 (-4.037651) | 3.768698 / 3.745712 (0.022986) | 1.924441 / 5.269862 (-3.345420) | 1.146364 / 4.565676 (-3.419312) | 0.067466 / 0.424275 (-0.356809) | 0.011175 / 0.007607 (0.003568) | 0.540978 / 0.226044 (0.314933) | 5.393120 / 2.268929 (3.124191) | 2.639027 / 55.444624 (-52.805597) | 2.327216 / 6.876477 (-4.549261) | 2.500532 / 2.142072 (0.358460) | 0.679120 / 4.805227 (-4.126107) | 0.148824 / 6.500664 (-6.351840) | 0.064195 / 0.075469 (-0.011274) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.158387 / 1.841788 (-0.683401) | 14.880751 / 8.074308 (6.806443) | 14.725249 / 10.191392 (4.533857) | 0.149785 / 0.680424 (-0.530639) | 0.017338 / 0.534201 (-0.516863) | 0.390980 / 0.579283 (-0.188303) | 0.425611 / 0.434364 (-0.008753) | 0.458851 / 0.540337 (-0.081487) | 0.559209 / 1.386936 (-0.827727) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006835 / 0.011353 (-0.004518) | 0.004318 / 0.011008 (-0.006690) | 0.076715 / 0.038508 (0.038207) | 0.033528 / 0.023109 (0.010419) | 0.411986 / 0.275898 (0.136087) | 0.438752 / 0.323480 (0.115272) | 0.004039 / 0.007986 (-0.003947) | 0.003509 / 0.004328 (-0.000819) | 0.077924 / 0.004250 (0.073673) | 0.049519 / 0.037052 (0.012467) | 0.420595 / 0.258489 (0.162106) | 0.450536 / 0.293841 (0.156695) | 0.032817 / 0.128546 (-0.095729) | 0.008963 / 0.075646 (-0.066684) | 0.083818 / 0.419271 (-0.335454) | 0.057591 / 0.043533 (0.014058) | 0.404605 / 0.255139 (0.149466) | 0.423661 / 0.283200 (0.140462) | 0.110698 / 0.141683 (-0.030984) | 1.512515 / 1.452155 (0.060361) | 1.569207 / 1.492716 (0.076490) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.200795 / 0.018006 (0.182789) | 0.448853 / 0.000490 (0.448363) | 0.003657 / 0.000200 (0.003457) | 0.000102 / 0.000054 (0.000048) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031612 / 0.037411 (-0.005799) | 0.116712 / 0.014526 (0.102186) | 0.126162 / 0.176557 (-0.050395) | 0.180522 / 0.737135 (-0.556614) | 0.129768 / 0.296338 (-0.166570) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.433797 / 0.215209 (0.218588) | 4.353099 / 2.077655 (2.275444) | 2.117582 / 1.504120 (0.613462) | 1.934487 / 1.541195 (0.393292) | 2.016988 / 1.468490 (0.548498) | 0.531387 / 4.584777 (-4.053390) | 3.843520 / 3.745712 (0.097807) | 1.879560 / 5.269862 (-3.390301) | 1.129445 / 4.565676 (-3.436231) | 0.065952 / 0.424275 (-0.358323) | 0.011566 / 0.007607 (0.003959) | 0.533949 / 0.226044 (0.307904) | 5.327447 / 2.268929 (3.058518) | 2.572202 / 55.444624 (-52.872422) | 2.240723 / 6.876477 (-4.635753) | 2.329290 / 2.142072 (0.187217) | 0.662162 / 4.805227 (-4.143066) | 0.143191 / 6.500664 (-6.357473) | 0.065273 / 0.075469 (-0.010196) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.274945 / 1.841788 (-0.566843) | 15.444511 / 8.074308 (7.370203) | 14.793524 / 10.191392 (4.602132) | 0.175607 / 0.680424 (-0.504817) | 0.017324 / 0.534201 (-0.516877) | 0.396172 / 0.579283 (-0.183111) | 0.437334 / 0.434364 (0.002970) | 0.472621 / 0.540337 (-0.067716) | 0.574888 / 1.386936 (-0.812048) |

![](https://cml.dev/watermark.png#b4ab1b3ed7257b0e0ad075d7271a51835f320a5e "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006976 / 0.011353 (-0.004377) | 0.004541 / 0.011008 (-0.006467) | 0.106085 / 0.038508 (0.067577) | 0.029148 / 0.023109 (0.006039) | 0.306386 / 0.275898 (0.030488) | 0.351474 / 0.323480 (0.027994) | 0.003924 / 0.007986 (-0.004062) | 0.004588 / 0.004328 (0.000260) | 0.090479 / 0.004250 (0.086229) | 0.041195 / 0.037052 (0.004142) | 0.346020 / 0.258489 (0.087531) | 0.362526 / 0.293841 (0.068685) | 0.041020 / 0.128546 (-0.087526) | 0.012536 / 0.075646 (-0.063110) | 0.333247 / 0.419271 (-0.086024) | 0.059786 / 0.043533 (0.016253) | 0.318094 / 0.255139 (0.062955) | 0.343879 / 0.283200 (0.060679) | 0.110083 / 0.141683 (-0.031600) | 1.514027 / 1.452155 (0.061872) | 1.551435 / 1.492716 (0.058719) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.235401 / 0.018006 (0.217395) | 0.544292 / 0.000490 (0.543803) | 0.005284 / 0.000200 (0.005084) | 0.000112 / 0.000054 (0.000057) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025008 / 0.037411 (-0.012403) | 0.102235 / 0.014526 (0.087709) | 0.105523 / 0.176557 (-0.071034) | 0.180846 / 0.737135 (-0.556289) | 0.107078 / 0.296338 (-0.189261) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.502374 / 0.215209 (0.287165) | 5.224254 / 2.077655 (3.146600) | 1.987193 / 1.504120 (0.483073) | 1.694680 / 1.541195 (0.153485) | 1.663907 / 1.468490 (0.195417) | 0.786470 / 4.584777 (-3.798307) | 4.977895 / 3.745712 (1.232183) | 4.713451 / 5.269862 (-0.556410) | 2.298763 / 4.565676 (-2.266913) | 0.090225 / 0.424275 (-0.334051) | 0.011427 / 0.007607 (0.003820) | 0.640686 / 0.226044 (0.414641) | 6.351727 / 2.268929 (4.082798) | 2.636912 / 55.444624 (-52.807712) | 2.075566 / 6.876477 (-4.800911) | 2.080260 / 2.142072 (-0.061812) | 0.952727 / 4.805227 (-3.852500) | 0.188651 / 6.500664 (-6.312013) | 0.068997 / 0.075469 (-0.006472) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.258878 / 1.841788 (-0.582910) | 15.444724 / 8.074308 (7.370416) | 17.521918 / 10.191392 (7.330526) | 0.189732 / 0.680424 (-0.490692) | 0.031084 / 0.534201 (-0.503117) | 0.445150 / 0.579283 (-0.134133) | 0.575844 / 0.434364 (0.141480) | 0.498162 / 0.540337 (-0.042176) | 0.635885 / 1.386936 (-0.751051) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007402 / 0.011353 (-0.003951) | 0.005058 / 0.011008 (-0.005950) | 0.077659 / 0.038508 (0.039151) | 0.034934 / 0.023109 (0.011825) | 0.373139 / 0.275898 (0.097241) | 0.411857 / 0.323480 (0.088377) | 0.003751 / 0.007986 (-0.004235) | 0.003634 / 0.004328 (-0.000695) | 0.075914 / 0.004250 (0.071663) | 0.037555 / 0.037052 (0.000503) | 0.387482 / 0.258489 (0.128993) | 0.434407 / 0.293841 (0.140566) | 0.040540 / 0.128546 (-0.088006) | 0.013458 / 0.075646 (-0.062189) | 0.096129 / 0.419271 (-0.323143) | 0.055369 / 0.043533 (0.011836) | 0.386564 / 0.255139 (0.131425) | 0.410417 / 0.283200 (0.127218) | 0.093265 / 0.141683 (-0.048418) | 1.432841 / 1.452155 (-0.019314) | 1.533180 / 1.492716 (0.040463) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.281051 / 0.018006 (0.263045) | 0.547635 / 0.000490 (0.547146) | 0.004434 / 0.000200 (0.004234) | 0.000105 / 0.000054 (0.000050) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026409 / 0.037411 (-0.011002) | 0.098586 / 0.014526 (0.084060) | 0.109223 / 0.176557 (-0.067334) | 0.165958 / 0.737135 (-0.571177) | 0.111751 / 0.296338 (-0.184587) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.542717 / 0.215209 (0.327508) | 5.530075 / 2.077655 (3.452420) | 2.351141 / 1.504120 (0.847022) | 2.021659 / 1.541195 (0.480464) | 1.964900 / 1.468490 (0.496410) | 0.819698 / 4.584777 (-3.765079) | 4.917412 / 3.745712 (1.171700) | 2.425149 / 5.269862 (-2.844712) | 1.561953 / 4.565676 (-3.003724) | 0.098417 / 0.424275 (-0.325858) | 0.012594 / 0.007607 (0.004986) | 0.717212 / 0.226044 (0.491168) | 6.994833 / 2.268929 (4.725904) | 2.997347 / 55.444624 (-52.447277) | 2.388366 / 6.876477 (-4.488111) | 2.502913 / 2.142072 (0.360841) | 1.030545 / 4.805227 (-3.774682) | 0.184844 / 6.500664 (-6.315820) | 0.076889 / 0.075469 (0.001420) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.371647 / 1.841788 (-0.470141) | 15.522995 / 8.074308 (7.448687) | 17.349823 / 10.191392 (7.158431) | 0.229709 / 0.680424 (-0.450714) | 0.023303 / 0.534201 (-0.510898) | 0.413874 / 0.579283 (-0.165409) | 0.567552 / 0.434364 (0.133188) | 0.491722 / 0.540337 (-0.048615) | 0.590640 / 1.386936 (-0.796296) |

![](https://cml.dev/watermark.png#f1911ffa5d1f58f509d04fe1ddeb9d00a63f94d5 "CML watermark")

following https://github.com/huggingface/datasets/pull/5944


What is the Right way to use discofuse dataset??

Thanks for opening https://huggingface.co/datasets/discofuse/discussions/3, let's continue the discussion over there if you don't mind

I have posted there also sir, please check
@lhoestq

[Click here for Dataset link](https://huggingface.co/datasets/discofuse/viewer/discofuse-wikipedia/train?row=6)
**Below is the following way, as per my understanding , Is it correct :question: :question:**

The **columns/features from `DiscoFuse dataset`** that will be the **input to the `encoder` and `decoder`** are:

[Click here for Dataset link](https://huggingface.co/datasets/discofuse/viewer/discofuse-wikipedia/train?row=6)

1. **coherent_first_sentence**

2. **coherent_second_sentence**

3. **incoherent_first_sentence**

4. **incoherent_second_sentence**

[Click here for Dataset link](https://huggingface.co/datasets/discofuse/viewer/discofuse-wikipedia/train?row=6)

The **`encoder` will take these four columns as input and encode them into a sequence of hidden states. The `decoder` will then take these hidden states as input and decode them into a new sentence that fuses the two original sentences together.**

The **discourse type, connective_string, has_coref_type_pronoun, and has_coref_type_nominal columns will not be used as input to the encoder or decoder.** These columns are used to provide additional information about the dataset, but they are not necessary for the task of sentence fusion.

Please correct me if I am wrong; otherwise, if this understanding is right, how shall I implement this task practically?


Support for data with instance-wise dictionary as features

enhancement

New feature or request

Hi ! We use the Arrow columnar format under the hood, which doesn't support such dictionaries: each field must have a fixed type and exist in each sample.

Instead you can restructure your data like
```
{
    "index": 0,
    "keys": ["2 * x + y >= 3"],
    "values": [["2 * x + y >= 3", "4 * x + 2 * y >= 6"]],
    }
},
...
{
    "index": 9999,
    "keys": ["x >= 6"],
    "values": [["x >= 6", "x >= 0", "x >= -1"]],
},
...
```

### Feature request

I notice that when loading data instances with feature type of python dictionary, the dictionary keys would be broadcast so that every instance has the same set of keys. Please see an example in the Motivation section.

It is possible to avoid this behavior, i.e., load dictionary features as it is and do not broadcast the keys among instances? Please note that these dictionaries would have to be processed dynamically at each training iteration into strings (and tokenized).

### Motivation

I am trying to load a dataset from a json file. Each instance of the dataset has a feature that is a dictionary but its keys depend on the instance. Every two instances may have different keys. For example, imagine a dataset that contains a set of math expressions from a bunch of mutually redundant expressions:
```
{
    "index": 0,
    "feature": {
        "2 * x + y >= 3": ["2 * x + y >= 3", "4 * x + 2 * y >= 6"],
        ...
    }
},
...
{
    "index": 9999,
    "feature": {
        "x >= 6": ["x >= 6", "x >= 0", "x >= -1"],
        ...
    }
},
...
```
When directly loading the dataset using `data = load_dataset("json", data_files=file_paths, split='train')`, each instance would have all the keys from other instances and None as values. That is, instance of index 0 becomes:
```
{
    "index": 0,
    "feature": {
        "2 * x + y >= 3": ["2 * x + y >= 3", "4 * x + 2 * y >= 6"],
        ...
        "x >= 6": None,  # keys from other instances
        ...
    }
},
```
This is not desirable. Moreover, issue would be raised if I attempt to combine two such datasets using `data = concatenate_datasets(multi_datasets)`, perhaps because their dictionary features contain different keys.

A solution I can think of is to store the dictionary features as a long string, and evaluate it later. Please kindly suggest any other solution using existing methods of datasets.

### Your contribution

N/A