[docs] Custom decoding transforms

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5836). All of your documentation changes will be reflected on that endpoint.

The error seems unrelated to the changes, so feel free to merge.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006562 / 0.011353 (-0.004791) | 0.004568 / 0.011008 (-0.006440) | 0.098151 / 0.038508 (0.059643) | 0.028117 / 0.023109 (0.005008) | 0.305442 / 0.275898 (0.029544) | 0.338288 / 0.323480 (0.014808) | 0.005012 / 0.007986 (-0.002973) | 0.003415 / 0.004328 (-0.000913) | 0.075022 / 0.004250 (0.070771) | 0.036869 / 0.037052 (-0.000183) | 0.301427 / 0.258489 (0.042937) | 0.348485 / 0.293841 (0.054644) | 0.030761 / 0.128546 (-0.097785) | 0.011461 / 0.075646 (-0.064185) | 0.321987 / 0.419271 (-0.097285) | 0.042885 / 0.043533 (-0.000648) | 0.300691 / 0.255139 (0.045552) | 0.333208 / 0.283200 (0.050008) | 0.090203 / 0.141683 (-0.051480) | 1.459744 / 1.452155 (0.007590) | 1.522960 / 1.492716 (0.030243) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.213219 / 0.018006 (0.195213) | 0.408118 / 0.000490 (0.407629) | 0.003716 / 0.000200 (0.003516) | 0.000077 / 0.000054 (0.000022) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023060 / 0.037411 (-0.014351) | 0.097423 / 0.014526 (0.082897) | 0.103988 / 0.176557 (-0.072568) | 0.162793 / 0.737135 (-0.574343) | 0.108282 / 0.296338 (-0.188056) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.431628 / 0.215209 (0.216419) | 4.300881 / 2.077655 (2.223226) | 2.058853 / 1.504120 (0.554733) | 1.897910 / 1.541195 (0.356715) | 1.991723 / 1.468490 (0.523233) | 0.699686 / 4.584777 (-3.885091) | 3.395004 / 3.745712 (-0.350708) | 1.841613 / 5.269862 (-3.428248) | 1.152347 / 4.565676 (-3.413330) | 0.082517 / 0.424275 (-0.341758) | 0.012323 / 0.007607 (0.004715) | 0.535812 / 0.226044 (0.309767) | 5.374103 / 2.268929 (3.105174) | 2.429662 / 55.444624 (-53.014962) | 2.097199 / 6.876477 (-4.779277) | 2.172625 / 2.142072 (0.030552) | 0.810156 / 4.805227 (-3.995071) | 0.151629 / 6.500664 (-6.349035) | 0.066528 / 0.075469 (-0.008941) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.220667 / 1.841788 (-0.621121) | 13.696976 / 8.074308 (5.622668) | 14.042916 / 10.191392 (3.851524) | 0.129626 / 0.680424 (-0.550798) | 0.016593 / 0.534201 (-0.517607) | 0.383747 / 0.579283 (-0.195536) | 0.386872 / 0.434364 (-0.047492) | 0.456524 / 0.540337 (-0.083813) | 0.545033 / 1.386936 (-0.841903) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006361 / 0.011353 (-0.004992) | 0.004516 / 0.011008 (-0.006493) | 0.077155 / 0.038508 (0.038647) | 0.027239 / 0.023109 (0.004130) | 0.359892 / 0.275898 (0.083994) | 0.391994 / 0.323480 (0.068514) | 0.004950 / 0.007986 (-0.003036) | 0.003379 / 0.004328 (-0.000949) | 0.077057 / 0.004250 (0.072806) | 0.039562 / 0.037052 (0.002509) | 0.364244 / 0.258489 (0.105755) | 0.416033 / 0.293841 (0.122192) | 0.031049 / 0.128546 (-0.097497) | 0.011479 / 0.075646 (-0.064167) | 0.086479 / 0.419271 (-0.332793) | 0.039381 / 0.043533 (-0.004151) | 0.372143 / 0.255139 (0.117004) | 0.388569 / 0.283200 (0.105369) | 0.090954 / 0.141683 (-0.050728) | 1.540957 / 1.452155 (0.088802) | 1.596841 / 1.492716 (0.104125) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.221130 / 0.018006 (0.203123) | 0.403728 / 0.000490 (0.403238) | 0.003172 / 0.000200 (0.002972) | 0.000078 / 0.000054 (0.000024) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024963 / 0.037411 (-0.012449) | 0.101065 / 0.014526 (0.086539) | 0.110846 / 0.176557 (-0.065710) | 0.158578 / 0.737135 (-0.578557) | 0.112235 / 0.296338 (-0.184104) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.457320 / 0.215209 (0.242111) | 4.548094 / 2.077655 (2.470439) | 2.175376 / 1.504120 (0.671256) | 1.964755 / 1.541195 (0.423561) | 2.008128 / 1.468490 (0.539638) | 0.702448 / 4.584777 (-3.882329) | 3.437595 / 3.745712 (-0.308117) | 3.009871 / 5.269862 (-2.259990) | 1.558181 / 4.565676 (-3.007496) | 0.082568 / 0.424275 (-0.341707) | 0.012371 / 0.007607 (0.004764) | 0.550688 / 0.226044 (0.324644) | 5.534210 / 2.268929 (3.265282) | 2.649605 / 55.444624 (-52.795020) | 2.317293 / 6.876477 (-4.559184) | 2.351525 / 2.142072 (0.209453) | 0.808971 / 4.805227 (-3.996256) | 0.152737 / 6.500664 (-6.347927) | 0.068416 / 0.075469 (-0.007053) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.340219 / 1.841788 (-0.501569) | 13.903388 / 8.074308 (5.829080) | 13.063477 / 10.191392 (2.872085) | 0.130216 / 0.680424 (-0.550208) | 0.016522 / 0.534201 (-0.517679) | 0.398946 / 0.579283 (-0.180337) | 0.382450 / 0.434364 (-0.051914) | 0.491007 / 0.540337 (-0.049330) | 0.577747 / 1.386936 (-0.809189) |

![](https://cml.dev/watermark.png#15c37ed142e4fbcb8c00ae62d4c71c84ce41959a "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007812 / 0.011353 (-0.003541) | 0.005563 / 0.011008 (-0.005446) | 0.099372 / 0.038508 (0.060864) | 0.035629 / 0.023109 (0.012520) | 0.301457 / 0.275898 (0.025559) | 0.339136 / 0.323480 (0.015656) | 0.006152 / 0.007986 (-0.001834) | 0.005843 / 0.004328 (0.001515) | 0.075280 / 0.004250 (0.071030) | 0.052789 / 0.037052 (0.015736) | 0.301805 / 0.258489 (0.043316) | 0.347918 / 0.293841 (0.054078) | 0.036182 / 0.128546 (-0.092364) | 0.012655 / 0.075646 (-0.062991) | 0.334428 / 0.419271 (-0.084844) | 0.062746 / 0.043533 (0.019213) | 0.296932 / 0.255139 (0.041793) | 0.314115 / 0.283200 (0.030916) | 0.121291 / 0.141683 (-0.020392) | 1.453252 / 1.452155 (0.001097) | 1.564714 / 1.492716 (0.071997) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.243810 / 0.018006 (0.225804) | 0.547129 / 0.000490 (0.546640) | 0.004666 / 0.000200 (0.004466) | 0.000089 / 0.000054 (0.000035) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028214 / 0.037411 (-0.009197) | 0.108878 / 0.014526 (0.094352) | 0.122313 / 0.176557 (-0.054243) | 0.182412 / 0.737135 (-0.554723) | 0.127014 / 0.296338 (-0.169324) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.423946 / 0.215209 (0.208737) | 4.207112 / 2.077655 (2.129457) | 2.048658 / 1.504120 (0.544538) | 1.843593 / 1.541195 (0.302398) | 1.952426 / 1.468490 (0.483936) | 0.712098 / 4.584777 (-3.872679) | 3.824971 / 3.745712 (0.079258) | 3.507141 / 5.269862 (-1.762721) | 1.868866 / 4.565676 (-2.696810) | 0.087895 / 0.424275 (-0.336380) | 0.012783 / 0.007607 (0.005176) | 0.524087 / 0.226044 (0.298042) | 5.246498 / 2.268929 (2.977570) | 2.495944 / 55.444624 (-52.948680) | 2.126779 / 6.876477 (-4.749698) | 2.315545 / 2.142072 (0.173472) | 0.859546 / 4.805227 (-3.945681) | 0.173457 / 6.500664 (-6.327208) | 0.067483 / 0.075469 (-0.007986) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.173851 / 1.841788 (-0.667937) | 15.091913 / 8.074308 (7.017605) | 14.640035 / 10.191392 (4.448643) | 0.168498 / 0.680424 (-0.511926) | 0.017513 / 0.534201 (-0.516688) | 0.425770 / 0.579283 (-0.153513) | 0.434248 / 0.434364 (-0.000116) | 0.504204 / 0.540337 (-0.036134) | 0.616885 / 1.386936 (-0.770051) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007775 / 0.011353 (-0.003578) | 0.005153 / 0.011008 (-0.005855) | 0.075461 / 0.038508 (0.036953) | 0.034994 / 0.023109 (0.011885) | 0.372389 / 0.275898 (0.096491) | 0.397911 / 0.323480 (0.074431) | 0.006572 / 0.007986 (-0.001413) | 0.005549 / 0.004328 (0.001220) | 0.075101 / 0.004250 (0.070851) | 0.054014 / 0.037052 (0.016962) | 0.368964 / 0.258489 (0.110475) | 0.425353 / 0.293841 (0.131512) | 0.035546 / 0.128546 (-0.093001) | 0.012707 / 0.075646 (-0.062939) | 0.087418 / 0.419271 (-0.331853) | 0.046425 / 0.043533 (0.002893) | 0.363982 / 0.255139 (0.108843) | 0.376421 / 0.283200 (0.093221) | 0.105369 / 0.141683 (-0.036314) | 1.494408 / 1.452155 (0.042253) | 1.596783 / 1.492716 (0.104067) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.258780 / 0.018006 (0.240773) | 0.533373 / 0.000490 (0.532883) | 0.000432 / 0.000200 (0.000232) | 0.000058 / 0.000054 (0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030687 / 0.037411 (-0.006725) | 0.110231 / 0.014526 (0.095705) | 0.123738 / 0.176557 (-0.052819) | 0.171999 / 0.737135 (-0.565137) | 0.127673 / 0.296338 (-0.168665) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.448058 / 0.215209 (0.232849) | 4.459381 / 2.077655 (2.381726) | 2.234020 / 1.504120 (0.729900) | 2.038616 / 1.541195 (0.497421) | 2.123795 / 1.468490 (0.655305) | 0.702664 / 4.584777 (-3.882113) | 3.837133 / 3.745712 (0.091420) | 2.138574 / 5.269862 (-3.131287) | 1.375955 / 4.565676 (-3.189722) | 0.086996 / 0.424275 (-0.337280) | 0.012461 / 0.007607 (0.004854) | 0.557978 / 0.226044 (0.331934) | 5.648613 / 2.268929 (3.379685) | 2.777829 / 55.444624 (-52.666796) | 2.392424 / 6.876477 (-4.484052) | 2.482823 / 2.142072 (0.340750) | 0.851891 / 4.805227 (-3.953336) | 0.171335 / 6.500664 (-6.329329) | 0.065041 / 0.075469 (-0.010428) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.319697 / 1.841788 (-0.522091) | 15.748688 / 8.074308 (7.674380) | 13.397042 / 10.191392 (3.205650) | 0.166424 / 0.680424 (-0.514000) | 0.017755 / 0.534201 (-0.516446) | 0.424989 / 0.579283 (-0.154294) | 0.424705 / 0.434364 (-0.009659) | 0.494190 / 0.540337 (-0.046147) | 0.588315 / 1.386936 (-0.798622) |

![](https://cml.dev/watermark.png#15c37ed142e4fbcb8c00ae62d4c71c84ce41959a "CML watermark")

Adds custom decoding transform solution to the docs to fix #5782.


Always set nullable fields in the writer

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006640 / 0.011353 (-0.004713) | 0.004606 / 0.011008 (-0.006402) | 0.098870 / 0.038508 (0.060362) | 0.028201 / 0.023109 (0.005092) | 0.304396 / 0.275898 (0.028498) | 0.339804 / 0.323480 (0.016324) | 0.005011 / 0.007986 (-0.002974) | 0.003530 / 0.004328 (-0.000799) | 0.075223 / 0.004250 (0.070973) | 0.037922 / 0.037052 (0.000870) | 0.310273 / 0.258489 (0.051784) | 0.348324 / 0.293841 (0.054483) | 0.030181 / 0.128546 (-0.098365) | 0.011584 / 0.075646 (-0.064062) | 0.322637 / 0.419271 (-0.096635) | 0.043119 / 0.043533 (-0.000414) | 0.314514 / 0.255139 (0.059375) | 0.334384 / 0.283200 (0.051185) | 0.092551 / 0.141683 (-0.049132) | 1.496694 / 1.452155 (0.044539) | 1.555426 / 1.492716 (0.062710) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.205078 / 0.018006 (0.187072) | 0.399200 / 0.000490 (0.398710) | 0.004881 / 0.000200 (0.004681) | 0.000200 / 0.000054 (0.000146) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025042 / 0.037411 (-0.012369) | 0.101501 / 0.014526 (0.086975) | 0.107430 / 0.176557 (-0.069127) | 0.170107 / 0.737135 (-0.567028) | 0.111253 / 0.296338 (-0.185086) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.460358 / 0.215209 (0.245149) | 4.592037 / 2.077655 (2.514383) | 2.222612 / 1.504120 (0.718493) | 2.022804 / 1.541195 (0.481610) | 2.040824 / 1.468490 (0.572334) | 0.700485 / 4.584777 (-3.884292) | 3.427847 / 3.745712 (-0.317866) | 2.836916 / 5.269862 (-2.432946) | 1.505055 / 4.565676 (-3.060621) | 0.083206 / 0.424275 (-0.341069) | 0.046492 / 0.007607 (0.038885) | 0.555562 / 0.226044 (0.329518) | 5.563574 / 2.268929 (3.294645) | 2.635273 / 55.444624 (-52.809351) | 2.299377 / 6.876477 (-4.577100) | 2.394512 / 2.142072 (0.252440) | 0.809541 / 4.805227 (-3.995686) | 0.151814 / 6.500664 (-6.348850) | 0.067241 / 0.075469 (-0.008228) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.188396 / 1.841788 (-0.653392) | 13.714596 / 8.074308 (5.640288) | 14.076906 / 10.191392 (3.885514) | 0.143447 / 0.680424 (-0.536977) | 0.016514 / 0.534201 (-0.517687) | 0.383075 / 0.579283 (-0.196209) | 0.386997 / 0.434364 (-0.047367) | 0.441941 / 0.540337 (-0.098396) | 0.522145 / 1.386936 (-0.864791) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006266 / 0.011353 (-0.005086) | 0.004562 / 0.011008 (-0.006446) | 0.077472 / 0.038508 (0.038964) | 0.027596 / 0.023109 (0.004486) | 0.400498 / 0.275898 (0.124600) | 0.406728 / 0.323480 (0.083248) | 0.004745 / 0.007986 (-0.003241) | 0.003375 / 0.004328 (-0.000954) | 0.076645 / 0.004250 (0.072394) | 0.037756 / 0.037052 (0.000703) | 0.415183 / 0.258489 (0.156694) | 0.413758 / 0.293841 (0.119917) | 0.030624 / 0.128546 (-0.097922) | 0.011525 / 0.075646 (-0.064121) | 0.086033 / 0.419271 (-0.333238) | 0.039307 / 0.043533 (-0.004226) | 0.418192 / 0.255139 (0.163053) | 0.403152 / 0.283200 (0.119952) | 0.094141 / 0.141683 (-0.047542) | 1.459012 / 1.452155 (0.006857) | 1.546493 / 1.492716 (0.053777) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.239494 / 0.018006 (0.221488) | 0.420918 / 0.000490 (0.420428) | 0.000411 / 0.000200 (0.000211) | 0.000057 / 0.000054 (0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024525 / 0.037411 (-0.012886) | 0.099793 / 0.014526 (0.085267) | 0.105888 / 0.176557 (-0.070669) | 0.155912 / 0.737135 (-0.581223) | 0.109937 / 0.296338 (-0.186401) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.470108 / 0.215209 (0.254899) | 4.696390 / 2.077655 (2.618735) | 2.467841 / 1.504120 (0.963721) | 2.275012 / 1.541195 (0.733818) | 2.430736 / 1.468490 (0.962245) | 0.700442 / 4.584777 (-3.884335) | 3.458451 / 3.745712 (-0.287261) | 1.921120 / 5.269862 (-3.348742) | 1.183292 / 4.565676 (-3.382384) | 0.083985 / 0.424275 (-0.340290) | 0.012510 / 0.007607 (0.004903) | 0.589066 / 0.226044 (0.363022) | 5.896070 / 2.268929 (3.627141) | 2.935379 / 55.444624 (-52.509245) | 2.599524 / 6.876477 (-4.276953) | 2.663426 / 2.142072 (0.521354) | 0.812096 / 4.805227 (-3.993131) | 0.152559 / 6.500664 (-6.348105) | 0.066906 / 0.075469 (-0.008563) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.333341 / 1.841788 (-0.508446) | 14.441667 / 8.074308 (6.367359) | 14.754069 / 10.191392 (4.562677) | 0.155707 / 0.680424 (-0.524716) | 0.016983 / 0.534201 (-0.517218) | 0.389386 / 0.579283 (-0.189897) | 0.394106 / 0.434364 (-0.040258) | 0.447355 / 0.540337 (-0.092982) | 0.533142 / 1.386936 (-0.853794) |

![](https://cml.dev/watermark.png#99ee4467ce77f8f718159a535e237dd8790b5bed "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007801 / 0.011353 (-0.003552) | 0.004884 / 0.011008 (-0.006124) | 0.114754 / 0.038508 (0.076245) | 0.040427 / 0.023109 (0.017318) | 0.402064 / 0.275898 (0.126166) | 0.428830 / 0.323480 (0.105350) | 0.006429 / 0.007986 (-0.001556) | 0.004394 / 0.004328 (0.000066) | 0.087681 / 0.004250 (0.083431) | 0.053684 / 0.037052 (0.016632) | 0.399967 / 0.258489 (0.141478) | 0.445298 / 0.293841 (0.151457) | 0.033194 / 0.128546 (-0.095352) | 0.010288 / 0.075646 (-0.065359) | 0.390719 / 0.419271 (-0.028552) | 0.059311 / 0.043533 (0.015778) | 0.393651 / 0.255139 (0.138512) | 0.418395 / 0.283200 (0.135196) | 0.121494 / 0.141683 (-0.020189) | 1.735470 / 1.452155 (0.283315) | 1.820485 / 1.492716 (0.327769) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.012887 / 0.018006 (-0.005119) | 0.491652 / 0.000490 (0.491162) | 0.005481 / 0.000200 (0.005281) | 0.000127 / 0.000054 (0.000073) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030931 / 0.037411 (-0.006480) | 0.125212 / 0.014526 (0.110686) | 0.136004 / 0.176557 (-0.040552) | 0.201686 / 0.737135 (-0.535449) | 0.140181 / 0.296338 (-0.156157) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.475003 / 0.215209 (0.259794) | 4.743918 / 2.077655 (2.666263) | 2.149422 / 1.504120 (0.645302) | 1.925016 / 1.541195 (0.383821) | 2.061441 / 1.468490 (0.592951) | 0.619845 / 4.584777 (-3.964932) | 4.534691 / 3.745712 (0.788979) | 2.248198 / 5.269862 (-3.021664) | 1.409868 / 4.565676 (-3.155808) | 0.080265 / 0.424275 (-0.344010) | 0.014455 / 0.007607 (0.006848) | 0.597810 / 0.226044 (0.371765) | 5.845492 / 2.268929 (3.576564) | 2.729139 / 55.444624 (-52.715486) | 2.313879 / 6.876477 (-4.562598) | 2.418763 / 2.142072 (0.276690) | 0.748687 / 4.805227 (-4.056540) | 0.165278 / 6.500664 (-6.335387) | 0.076848 / 0.075469 (0.001379) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.416349 / 1.841788 (-0.425439) | 17.440903 / 8.074308 (9.366595) | 17.025733 / 10.191392 (6.834341) | 0.167428 / 0.680424 (-0.512995) | 0.020484 / 0.534201 (-0.513717) | 0.470273 / 0.579283 (-0.109010) | 0.494380 / 0.434364 (0.060016) | 0.566131 / 0.540337 (0.025794) | 0.690444 / 1.386936 (-0.696492) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007695 / 0.011353 (-0.003657) | 0.005551 / 0.011008 (-0.005457) | 0.087812 / 0.038508 (0.049304) | 0.039107 / 0.023109 (0.015998) | 0.436461 / 0.275898 (0.160563) | 0.465116 / 0.323480 (0.141636) | 0.006590 / 0.007986 (-0.001396) | 0.004672 / 0.004328 (0.000343) | 0.087109 / 0.004250 (0.082858) | 0.054227 / 0.037052 (0.017175) | 0.442660 / 0.258489 (0.184171) | 0.484296 / 0.293841 (0.190455) | 0.033308 / 0.128546 (-0.095238) | 0.010780 / 0.075646 (-0.064866) | 0.095255 / 0.419271 (-0.324016) | 0.054399 / 0.043533 (0.010866) | 0.431734 / 0.255139 (0.176595) | 0.453583 / 0.283200 (0.170383) | 0.116067 / 0.141683 (-0.025616) | 1.780701 / 1.452155 (0.328546) | 1.851077 / 1.492716 (0.358360) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.228000 / 0.018006 (0.209994) | 0.485733 / 0.000490 (0.485243) | 0.003955 / 0.000200 (0.003755) | 0.000109 / 0.000054 (0.000054) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033974 / 0.037411 (-0.003437) | 0.134504 / 0.014526 (0.119978) | 0.144421 / 0.176557 (-0.032135) | 0.202171 / 0.737135 (-0.534964) | 0.152015 / 0.296338 (-0.144323) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.520462 / 0.215209 (0.305253) | 5.233339 / 2.077655 (3.155684) | 2.575013 / 1.504120 (1.070893) | 2.384119 / 1.541195 (0.842924) | 2.403856 / 1.468490 (0.935366) | 0.618656 / 4.584777 (-3.966121) | 4.663582 / 3.745712 (0.917870) | 3.738594 / 5.269862 (-1.531268) | 1.794903 / 4.565676 (-2.770773) | 0.077903 / 0.424275 (-0.346372) | 0.014681 / 0.007607 (0.007074) | 0.648615 / 0.226044 (0.422570) | 6.503721 / 2.268929 (4.234792) | 3.326239 / 55.444624 (-52.118386) | 2.989791 / 6.876477 (-3.886685) | 2.995479 / 2.142072 (0.853407) | 0.765483 / 4.805227 (-4.039744) | 0.169783 / 6.500664 (-6.330882) | 0.077533 / 0.075469 (0.002064) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.518736 / 1.841788 (-0.323051) | 17.989119 / 8.074308 (9.914811) | 15.484365 / 10.191392 (5.292973) | 0.168507 / 0.680424 (-0.511917) | 0.020289 / 0.534201 (-0.513912) | 0.467491 / 0.579283 (-0.111793) | 0.501714 / 0.434364 (0.067350) | 0.553418 / 0.540337 (0.013081) | 0.662199 / 1.386936 (-0.724737) |

![](https://cml.dev/watermark.png#5ebda17e4362bd5f6123543a14fa526a3b54481a "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007044 / 0.011353 (-0.004309) | 0.004750 / 0.011008 (-0.006258) | 0.096694 / 0.038508 (0.058186) | 0.035682 / 0.023109 (0.012573) | 0.300613 / 0.275898 (0.024715) | 0.334831 / 0.323480 (0.011351) | 0.006428 / 0.007986 (-0.001558) | 0.004456 / 0.004328 (0.000128) | 0.075060 / 0.004250 (0.070810) | 0.053166 / 0.037052 (0.016114) | 0.299601 / 0.258489 (0.041112) | 0.359521 / 0.293841 (0.065680) | 0.028072 / 0.128546 (-0.100474) | 0.009216 / 0.075646 (-0.066430) | 0.328895 / 0.419271 (-0.090377) | 0.050881 / 0.043533 (0.007349) | 0.298265 / 0.255139 (0.043126) | 0.318095 / 0.283200 (0.034896) | 0.116046 / 0.141683 (-0.025637) | 1.491312 / 1.452155 (0.039157) | 1.556053 / 1.492716 (0.063337) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.014248 / 0.018006 (-0.003758) | 0.551455 / 0.000490 (0.550965) | 0.006096 / 0.000200 (0.005897) | 0.000145 / 0.000054 (0.000091) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030598 / 0.037411 (-0.006813) | 0.109549 / 0.014526 (0.095023) | 0.123207 / 0.176557 (-0.053350) | 0.181940 / 0.737135 (-0.555195) | 0.128965 / 0.296338 (-0.167374) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.404552 / 0.215209 (0.189343) | 4.030674 / 2.077655 (1.953020) | 1.841819 / 1.504120 (0.337699) | 1.650055 / 1.541195 (0.108860) | 1.763208 / 1.468490 (0.294718) | 0.532715 / 4.584777 (-4.052062) | 3.774810 / 3.745712 (0.029098) | 3.221927 / 5.269862 (-2.047934) | 1.607974 / 4.565676 (-2.957702) | 0.067160 / 0.424275 (-0.357116) | 0.012479 / 0.007607 (0.004872) | 0.498801 / 0.226044 (0.272757) | 4.980567 / 2.268929 (2.711638) | 2.356017 / 55.444624 (-53.088608) | 2.018975 / 6.876477 (-4.857502) | 2.218343 / 2.142072 (0.076270) | 0.645714 / 4.805227 (-4.159514) | 0.145470 / 6.500664 (-6.355195) | 0.065666 / 0.075469 (-0.009803) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.205756 / 1.841788 (-0.636031) | 15.682779 / 8.074308 (7.608470) | 14.748987 / 10.191392 (4.557595) | 0.167105 / 0.680424 (-0.513319) | 0.017554 / 0.534201 (-0.516647) | 0.393924 / 0.579283 (-0.185359) | 0.432659 / 0.434364 (-0.001705) | 0.502033 / 0.540337 (-0.038304) | 0.602244 / 1.386936 (-0.784692) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007077 / 0.011353 (-0.004276) | 0.004911 / 0.011008 (-0.006097) | 0.075120 / 0.038508 (0.036612) | 0.035460 / 0.023109 (0.012351) | 0.362569 / 0.275898 (0.086671) | 0.398995 / 0.323480 (0.075515) | 0.006587 / 0.007986 (-0.001398) | 0.004571 / 0.004328 (0.000242) | 0.074647 / 0.004250 (0.070397) | 0.057331 / 0.037052 (0.020279) | 0.365123 / 0.258489 (0.106634) | 0.408617 / 0.293841 (0.114776) | 0.028911 / 0.128546 (-0.099635) | 0.009533 / 0.075646 (-0.066113) | 0.081566 / 0.419271 (-0.337705) | 0.048841 / 0.043533 (0.005308) | 0.367245 / 0.255139 (0.112106) | 0.375975 / 0.283200 (0.092776) | 0.123211 / 0.141683 (-0.018472) | 1.471588 / 1.452155 (0.019433) | 1.569342 / 1.492716 (0.076625) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.328443 / 0.018006 (0.310436) | 0.541402 / 0.000490 (0.540912) | 0.000440 / 0.000200 (0.000240) | 0.000058 / 0.000054 (0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030772 / 0.037411 (-0.006639) | 0.115833 / 0.014526 (0.101307) | 0.127837 / 0.176557 (-0.048719) | 0.180897 / 0.737135 (-0.556238) | 0.132458 / 0.296338 (-0.163881) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.445979 / 0.215209 (0.230770) | 4.453101 / 2.077655 (2.375447) | 2.276625 / 1.504120 (0.772505) | 2.102167 / 1.541195 (0.560972) | 2.181583 / 1.468490 (0.713093) | 0.525069 / 4.584777 (-4.059708) | 3.803446 / 3.745712 (0.057734) | 1.954173 / 5.269862 (-3.315688) | 1.088734 / 4.565676 (-3.476942) | 0.066020 / 0.424275 (-0.358255) | 0.012158 / 0.007607 (0.004551) | 0.546828 / 0.226044 (0.320783) | 5.454060 / 2.268929 (3.185132) | 2.756154 / 55.444624 (-52.688470) | 2.476501 / 6.876477 (-4.399976) | 2.525875 / 2.142072 (0.383803) | 0.647515 / 4.805227 (-4.157712) | 0.144511 / 6.500664 (-6.356153) | 0.067060 / 0.075469 (-0.008409) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.306456 / 1.841788 (-0.535332) | 15.822623 / 8.074308 (7.748315) | 14.929114 / 10.191392 (4.737721) | 0.168650 / 0.680424 (-0.511773) | 0.018043 / 0.534201 (-0.516158) | 0.396712 / 0.579283 (-0.182572) | 0.425800 / 0.434364 (-0.008564) | 0.466452 / 0.540337 (-0.073885) | 0.564370 / 1.386936 (-0.822566) |

![](https://cml.dev/watermark.png#5ebda17e4362bd5f6123543a14fa526a3b54481a "CML watermark")

This fixes loading of e.g. parquet data with non-nullable fields.

Indeed `datasets.Features` doesn't support non-nullable fields, which can lead to data not concatenable due to arrow schema mismatch.


Is uint8 supported?

Hi ! The numpy formatting detaults to int64 and float32 - but you can use uint8 using
```python
ds = ds.with_format("numpy", dtype=np.uint8)
```

Thank you!
By setting `ds.with_format("numpy", dtype=np.uint8)`, the dataset returns the data in `uint8`.

However, `with_format` and `set_format` seem to cast the data on-the-fly.
I want to reduce the dataset size by using `uint8` instead of `int64` and I observe no difference between using `int64` and `uint8` for the vector.
Is there any way to actually store the data in `uint8` and save the disk space and the downloading time when loaded from the hub?

If the feature type is `Value("uint8")` then it's written an uint8 on disk using the uint8 Arrow dtype.

e.g.
```python
ds = Dataset.from_dict({"a": range(10)}, features=Features({"a": Value("uint8")}))
ds.data.nbytes
# 10
```

Oh, I understand now.
The data was stored in `uint8` from the beginning (when the dataset returns `int64`).

Thank you for your time!
My question is fully resolved.

### Describe the bug

I expect the dataset to store the data in the `uint8` data type, but it's returning `int64` instead.
While I've found that `datasets` doesn't yet support float16 (https://github.com/huggingface/datasets/issues/4981), I'm wondering if this is the case for other data types as well.
Is there a way to store vector data as `uint8` and then upload it to the hub?

### Steps to reproduce the bug

```python
from datasets import Features, Dataset, Sequence, Value
import numpy as np

dataset = Dataset.from_dict(
    {"vector": [np.array([0, 1, 2], dtype=np.uint8)]}, features=Features({"vector": Sequence(Value("uint8"))})
).with_format("numpy")

print(dataset[0]["vector"].dtype)
```

### Expected behavior

Expected: `uint8`
Actual: `int64`

### Environment info

- `datasets` version: 2.12.0
- Platform: macOS-12.1-x86_64-i386-64bit
- Python version: 3.8.12
- Huggingface_hub version: 0.12.1
- PyArrow version: 11.0.0
- Pandas version: 1.5.3


Unable to push dataset - `create_pr` problem

Thanks for reporting, @agombert.

In this case, I think the root issue is authentication: before pushing to Hub, you should authenticate. See our docs: https://huggingface.co/docs/datasets/upload_dataset#upload-with-python
> 2. To upload a dataset on the Hub in Python, you need to log in to your Hugging Face account:
   ```
   huggingface-cli login
   ```

Hey @albertvillanova well I actually did :D

<img width="1079" alt="Capture d’écran 2023-05-09 à 18 02 58" src="https://github.com/huggingface/datasets/assets/17645711/e091aa20-06b1-4dd3-bfdb-35e832c66f8d">

That is weird that you get a Forbidden error if you are properly authenticated...

Today we had a big outage issue affecting the Hugging Face Hub. Could you please retry to push_to_hub your dataset? Maybe that was the cause...

Yes I've just tried again and same error 403 :/

Login successful but also got this error "Forbidden: pass `create_pr=1` as a query parameter to create a Pull Request"

> Make sure your API token has a `write` role. I had the same issue as you with the `read` token. Creating a `write` token and using that solved the issue.

I generate a token with write role. It works! thank you so much.

### Describe the bug

I can't upload to the hub the dataset I manually created locally (Image dataset). I have a problem when using the method `.push_to_hub` which asks for a `create_pr` attribute which is not compatible.

### Steps to reproduce the bug

here what I have:

```python
dataset.push_to_hub("agomberto/FrenchCensus-handwritten-texts")
```
Output:
```python
Pushing split train to the Hub.
Pushing dataset shards to the dataset hub:   0%|                                                                                | 0/2 [00:00<?, ?it/s]
Creating parquet from Arrow format:   0%|                                                                                | 0/3 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 12.70ba/s]
Pushing dataset shards to the dataset hub:   0%|                                                                                | 0/2 [00:01<?, ?it/s]

---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
File ~/miniconda3/envs/hwocr/lib/python3.8/site-packages/huggingface_hub/utils/_errors.py:259, in hf_raise_for_status(response, endpoint_name)
    258 try:
--> 259     response.raise_for_status()
    260 except HTTPError as e:

File ~/miniconda3/envs/hwocr/lib/python3.8/site-packages/requests/models.py:1021, in Response.raise_for_status(self)
   1020 if http_error_msg:
-> 1021     raise HTTPError(http_error_msg, response=self)

HTTPError: 403 Client Error: Forbidden for url: https://huggingface.co/api/datasets/agomberto/FrenchCensus-handwritten-texts/commit/main

The above exception was the direct cause of the following exception:

HfHubHTTPError                            Traceback (most recent call last)
Cell In[7], line 1
----> 1 dataset.push_to_hub("agomberto/FrenchCensus-handwritten-texts")

File ~/miniconda3/envs/hwocr/lib/python3.8/site-packages/datasets/dataset_dict.py:1583, in DatasetDict.push_to_hub(self, repo_id, private, token, branch, max_shard_size, num_shards, embed_external_files)
   1581 logger.warning(f"Pushing split {split} to the Hub.")
   1582 # The split=key needs to be removed before merging
-> 1583 repo_id, split, uploaded_size, dataset_nbytes, _, _ = self[split]._push_parquet_shards_to_hub(
   1584     repo_id,
   1585     split=split,
   1586     private=private,
   1587     token=token,
   1588     branch=branch,
   1589     max_shard_size=max_shard_size,
   1590     num_shards=num_shards.get(split),
   1591     embed_external_files=embed_external_files,
   1592 )
   1593 total_uploaded_size += uploaded_size
   1594 total_dataset_nbytes += dataset_nbytes

File ~/miniconda3/envs/hwocr/lib/python3.8/site-packages/datasets/arrow_dataset.py:5275, in Dataset._push_parquet_shards_to_hub(self, repo_id, split, private, token, branch, max_shard_size, num_shards, embed_external_files)
   5273         shard.to_parquet(buffer)
   5274         uploaded_size += buffer.tell()
-> 5275         _retry(
   5276             api.upload_file,
   5277             func_kwargs={
   5278                 "path_or_fileobj": buffer.getvalue(),
   5279                 "path_in_repo": shard_path_in_repo,
   5280                 "repo_id": repo_id,
   5281                 "token": token,
   5282                 "repo_type": "dataset",
   5283                 "revision": branch,
   5284             },
   5285             exceptions=HTTPError,
   5286             status_codes=[504],
   5287             base_wait_time=2.0,
   5288             max_retries=5,
   5289             max_wait_time=20.0,
   5290         )
   5291     shards_path_in_repo.append(shard_path_in_repo)
   5293 # Cleanup to remove unused files

File ~/miniconda3/envs/hwocr/lib/python3.8/site-packages/datasets/utils/file_utils.py:285, in _retry(func, func_args, func_kwargs, exceptions, status_codes, max_retries, base_wait_time, max_wait_time)
    283 except exceptions as err:
    284     if retry >= max_retries or (status_codes and err.response.status_code not in status_codes):
--> 285         raise err
    286     else:
    287         sleep_time = min(max_wait_time, base_wait_time * 2**retry)  # Exponential backoff

File ~/miniconda3/envs/hwocr/lib/python3.8/site-packages/datasets/utils/file_utils.py:282, in _retry(func, func_args, func_kwargs, exceptions, status_codes, max_retries, base_wait_time, max_wait_time)
    280 while True:
    281     try:
--> 282         return func(*func_args, **func_kwargs)
    283     except exceptions as err:
    284         if retry >= max_retries or (status_codes and err.response.status_code not in status_codes):

File ~/miniconda3/envs/hwocr/lib/python3.8/site-packages/huggingface_hub/utils/_validators.py:120, in validate_hf_hub_args.<locals>._inner_fn(*args, **kwargs)
    117 if check_use_auth_token:
    118     kwargs = smoothly_deprecate_use_auth_token(fn_name=fn.__name__, has_token=has_token, kwargs=kwargs)
--> 120 return fn(*args, **kwargs)

File ~/miniconda3/envs/hwocr/lib/python3.8/site-packages/huggingface_hub/hf_api.py:2998, in HfApi.upload_file(self, path_or_fileobj, path_in_repo, repo_id, token, repo_type, revision, commit_message, commit_description, create_pr, parent_commit)
   2990 commit_message = (
   2991     commit_message if commit_message is not None else f"Upload {path_in_repo} with huggingface_hub"
   2992 )
   2993 operation = CommitOperationAdd(
   2994     path_or_fileobj=path_or_fileobj,
   2995     path_in_repo=path_in_repo,
   2996 )
-> 2998 commit_info = self.create_commit(
   2999     repo_id=repo_id,
   3000     repo_type=repo_type,
   3001     operations=[operation],
   3002     commit_message=commit_message,
   3003     commit_description=commit_description,
   3004     token=token,
   3005     revision=revision,
   3006     create_pr=create_pr,
   3007     parent_commit=parent_commit,
   3008 )
   3010 if commit_info.pr_url is not None:
   3011     revision = quote(_parse_revision_from_pr_url(commit_info.pr_url), safe="")

File ~/miniconda3/envs/hwocr/lib/python3.8/site-packages/huggingface_hub/utils/_validators.py:120, in validate_hf_hub_args.<locals>._inner_fn(*args, **kwargs)
    117 if check_use_auth_token:
    118     kwargs = smoothly_deprecate_use_auth_token(fn_name=fn.__name__, has_token=has_token, kwargs=kwargs)
--> 120 return fn(*args, **kwargs)

File ~/miniconda3/envs/hwocr/lib/python3.8/site-packages/huggingface_hub/hf_api.py:2548, in HfApi.create_commit(self, repo_id, operations, commit_message, commit_description, token, repo_type, revision, create_pr, num_threads, parent_commit)
   2546 try:
   2547     commit_resp = get_session().post(url=commit_url, headers=headers, data=data, params=params)
-> 2548     hf_raise_for_status(commit_resp, endpoint_name="commit")
   2549 except RepositoryNotFoundError as e:
   2550     e.append_to_message(_CREATE_COMMIT_NO_REPO_ERROR_MESSAGE)

File ~/miniconda3/envs/hwocr/lib/python3.8/site-packages/huggingface_hub/utils/_errors.py:301, in hf_raise_for_status(response, endpoint_name)
    297     raise BadRequestError(message, response=response) from e
    299 # Convert `HTTPError` into a `HfHubHTTPError` to display request information
    300 # as well (request id and/or server error message)
--> 301 raise HfHubHTTPError(str(e), response=response) from e

HfHubHTTPError: 403 Client Error: Forbidden for url: https://huggingface.co/api/datasets/agomberto/FrenchCensus-handwritten-texts/commit/main (Request ID: Root=1-645a66bf-255ad91602a6404e6cb70fba)

Forbidden: pass `create_pr=1` as a query parameter to create a Pull Request
```

And then when I do

```python
dataset.push_to_hub("agomberto/FrenchCensus-handwritten-texts", create_pr=1)
```

I get

```python
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[8], line 1
----> 1 dataset.push_to_hub("agomberto/FrenchCensus-handwritten-texts", create_pr=1)

TypeError: push_to_hub() got an unexpected keyword argument 'create_pr'
```

### Expected behavior

I would like to have the dataset updloaded [here](https://huggingface.co/datasets/agomberto/FrenchCensus-handwritten-texts).

### Environment info

```bash
- `datasets` version: 2.12.0
- Platform: macOS-13.3.1-arm64-arm-64bit
- Python version: 3.8.16
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 1.5.3
```


### Describe the bug

Running [Bert-Large-Cased](https://huggingface.co/bert-large-cased) model causes `HTTPError`, with the following traceback-

```
HTTPError                                 Traceback (most recent call last)
<ipython-input-6-5c580443a1ad> in <module>
----> 1 tokenizer = BertTokenizer.from_pretrained('bert-large-cased')

~/miniconda3/envs/cmd-chall/lib/python3.7/site-packages/transformers/tokenization_utils_base.py in from_pretrained(cls, pretrained_model_name_or_path, *init_inputs, **kwargs)
   1646             # At this point pretrained_model_name_or_path is either a directory or a model identifier name
   1647             fast_tokenizer_file = get_fast_tokenizer_file(
-> 1648                 pretrained_model_name_or_path, revision=revision, use_auth_token=use_auth_token
   1649             )
   1650             additional_files_names = {

~/miniconda3/envs/cmd-chall/lib/python3.7/site-packages/transformers/tokenization_utils_base.py in get_fast_tokenizer_file(path_or_repo, revision, use_auth_token)
   3406     """
   3407     # Inspect all files from the repo/folder.
-> 3408     all_files = get_list_of_files(path_or_repo, revision=revision, use_auth_token=use_auth_token)
   3409     tokenizer_files_map = {}
   3410     for file_name in all_files:

~/miniconda3/envs/cmd-chall/lib/python3.7/site-packages/transformers/file_utils.py in get_list_of_files(path_or_repo, revision, use_auth_token)
   1685         token = None
   1686     model_info = HfApi(endpoint=HUGGINGFACE_CO_RESOLVE_ENDPOINT).model_info(
-> 1687         path_or_repo, revision=revision, token=token
   1688     )
   1689     return [f.rfilename for f in model_info.siblings]

~/miniconda3/envs/cmd-chall/lib/python3.7/site-packages/huggingface_hub/hf_api.py in model_info(self, repo_id, revision, token)
    246         )
    247         r = requests.get(path, headers=headers)
--> 248         r.raise_for_status()
    249         d = r.json()
    250         return ModelInfo(**d)

~/miniconda3/envs/cmd-chall/lib/python3.7/site-packages/requests/models.py in raise_for_status(self)
    951
    952         if http_error_msg:
--> 953             raise HTTPError(http_error_msg, response=self)
    954
    955     def close(self):

HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/api/models/bert-large-cased
```

I have also tried running in offline mode, as [discussed here](https://huggingface.co/docs/transformers/installation#offline-mode)
```
HF_DATASETS_OFFLINE=1
TRANSFORMERS_OFFLINE=1
```

### Steps to reproduce the bug

1.  `from transformers import BertTokenizer, BertModel`
2.  `tokenizer = BertTokenizer.from_pretrained('bert-large-cased')`

### Expected behavior

Run without the HTTP error.

### Environment info

| # Name             | Version    | Build                       | Channel |   |
|--------------------|------------|-----------------------------|---------|---|
| _libgcc_mutex      | 0.1        | main                        |         |   |
| _openmp_mutex      | 4.5        | 1_gnu                       |         |   |
| _pytorch_select    | 0.1        | cpu_0                       |         |   |
| appdirs            | 1.4.4      | pypi_0                      | pypi    |   |
| backcall           | 0.2.0      | pypi_0                      | pypi    |   |
| blas               | 1.0        | mkl                         |         |   |
| bzip2              | 1.0.8      | h7b6447c_0                  |         |   |
| ca-certificates    | 2021.7.5   | h06a4308_1                  |         |   |
| certifi            | 2021.5.30  | py37h06a4308_0              |         |   |
| cffi               | 1.14.6     | py37h400218f_0              |         |   |
| charset-normalizer | 2.0.3      | pypi_0                      | pypi    |   |
| click              | 8.0.1      | pypi_0                      | pypi    |   |
| colorama           | 0.4.4      | pypi_0                      | pypi    |   |
| cudatoolkit        | 11.1.74    | h6bb024c_0                  | nvidia  |   |
| cycler             | 0.11.0     | pypi_0                      | pypi    |   |
| decorator          | 5.0.9      | pypi_0                      | pypi    |   |
| docker-pycreds     | 0.4.0      | pypi_0                      | pypi    |   |
| docopt             | 0.6.2      | pypi_0                      | pypi    |   |
| dominate           | 2.6.0      | pypi_0                      | pypi    |   |
| ffmpeg             | 4.3        | hf484d3e_0                  | pytorch |   |
| filelock           | 3.0.12     | pypi_0                      | pypi    |   |
| fonttools          | 4.38.0     | pypi_0                      | pypi    |   |
| freetype           | 2.10.4     | h5ab3b9f_0                  |         |   |
| gitdb              | 4.0.7      | pypi_0                      | pypi    |   |
| gitpython          | 3.1.18     | pypi_0                      | pypi    |   |
| gmp                | 6.2.1      | h2531618_2                  |         |   |
| gnutls             | 3.6.15     | he1e5248_0                  |         |   |
| huggingface-hub    | 0.0.12     | pypi_0                      | pypi    |   |
| humanize           | 3.10.0     | pypi_0                      | pypi    |   |
| idna               | 3.2        | pypi_0                      | pypi    |   |
| importlib-metadata | 4.6.1      | pypi_0                      | pypi    |   |
| intel-openmp       | 2019.4     | 243                         |         |   |
| ipdb               | 0.13.9     | pypi_0                      | pypi    |   |
| ipython            | 7.25.0     | pypi_0                      | pypi    |   |
| ipython-genutils   | 0.2.0      | pypi_0                      | pypi    |   |
| jedi               | 0.18.0     | pypi_0                      | pypi    |   |
| joblib             | 1.0.1      | pypi_0                      | pypi    |   |
| jpeg               | 9b         | h024ee3a_2                  |         |   |
| jsonpickle         | 1.5.2      | pypi_0                      | pypi    |   |
| kiwisolver         | 1.4.4      | pypi_0                      | pypi    |   |
| lame               | 3.100      | h7b6447c_0                  |         |   |
| lcms2              | 2.12       | h3be6417_0                  |         |   |
| ld_impl_linux-64   | 2.35.1     | h7274673_9                  |         |   |
| libffi             | 3.3        | he6710b0_2                  |         |   |
| libgcc-ng          | 9.3.0      | h5101ec6_17                 |         |   |
| libgomp            | 9.3.0      | h5101ec6_17                 |         |   |
| libiconv           | 1.15       | h63c8f33_5                  |         |   |
| libidn2            | 2.3.2      | h7f8727e_0                  |         |   |
| libmklml           | 2019.0.5   | 0                           |         |   |
| libpng             | 1.6.37     | hbc83047_0                  |         |   |
| libstdcxx-ng       | 9.3.0      | hd4cf53a_17                 |         |   |
| libtasn1           | 4.16.0     | h27cfd23_0                  |         |   |
| libtiff            | 4.2.0      | h85742a9_0                  |         |   |
| libunistring       | 0.9.10     | h27cfd23_0                  |         |   |
| libuv              | 1.40.0     | h7b6447c_0                  |         |   |
| libwebp-base       | 1.2.0      | h27cfd23_0                  |         |   |
| lz4-c              | 1.9.3      | h2531618_0                  |         |   |
| matplotlib         | 3.5.3      | pypi_0                      | pypi    |   |
| matplotlib-inline  | 0.1.2      | pypi_0                      | pypi    |   |
| mergedeep          | 1.3.4      | pypi_0                      | pypi    |   |
| mkl                | 2020.2     | 256                         |         |   |
| mkl-service        | 2.3.0      | py37he8ac12f_0              |         |   |
| mkl_fft            | 1.3.0      | py37h54f3939_0              |         |   |
| mkl_random         | 1.1.1      | py37h0573a6f_0              |         |   |
| msgpack            | 1.0.2      | pypi_0                      | pypi    |   |
| munch              | 2.5.0      | pypi_0                      | pypi    |   |
| ncurses            | 6.2        | he6710b0_1                  |         |   |
| nettle             | 3.7.3      | hbbd107a_1                  |         |   |
| ninja              | 1.10.2     | hff7bd54_1                  |         |   |
| nltk               | 3.8.1      | pypi_0                      | pypi    |   |
| numpy              | 1.19.2     | py37h54aff64_0              |         |   |
| numpy-base         | 1.19.2     | py37hfa32c7d_0              |         |   |
| olefile            | 0.46       | py37_0                      |         |   |
| openh264           | 2.1.0      | hd408876_0                  |         |   |
| openjpeg           | 2.3.0      | h05c96fa_1                  |         |   |
| openssl            | 1.1.1k     | h27cfd23_0                  |         |   |
| packaging          | 21.0       | pypi_0                      | pypi    |   |
| pandas             | 1.3.1      | pypi_0                      | pypi    |   |
| parso              | 0.8.2      | pypi_0                      | pypi    |   |
| pathtools          | 0.1.2      | pypi_0                      | pypi    |   |
| pexpect            | 4.8.0      | pypi_0                      | pypi    |   |
| pickleshare        | 0.7.5      | pypi_0                      | pypi    |   |
| pillow             | 8.3.1      | py37h2c7a002_0              |         |   |
| pip                | 21.1.3     | py37h06a4308_0              |         |   |
| prompt-toolkit     | 3.0.19     | pypi_0                      | pypi    |   |
| protobuf           | 4.21.12    | pypi_0                      | pypi    |   |
| psutil             | 5.8.0      | pypi_0                      | pypi    |   |
| ptyprocess         | 0.7.0      | pypi_0                      | pypi    |   |
| py-cpuinfo         | 8.0.0      | pypi_0                      | pypi    |   |
| pycparser          | 2.20       | py_2                        |         |   |
| pygments           | 2.9.0      | pypi_0                      | pypi    |   |
| pyparsing          | 2.4.7      | pypi_0                      | pypi    |   |
| python             | 3.7.10     | h12debd9_4                  |         |   |
| python-dateutil    | 2.8.2      | pypi_0                      | pypi    |   |
| pytorch            | 1.9.0      | py3.7_cuda11.1_cudnn8.0.5_0 | pytorch |   |
| pytz               | 2021.1     | pypi_0                      | pypi    |   |
| pyyaml             | 5.4.1      | pypi_0                      | pypi    |   |
| readline           | 8.1        | h27cfd23_0                  |         |   |
| regex              | 2022.10.31 | pypi_0                      | pypi    |   |
| requests           | 2.26.0     | pypi_0                      | pypi    |   |
| sacred             | 0.8.2      | pypi_0                      | pypi    |   |
| sacremoses         | 0.0.45     | pypi_0                      | pypi    |   |
| scikit-learn       | 0.24.2     | pypi_0                      | pypi    |   |
| scipy              | 1.7.0      | pypi_0                      | pypi    |   |
| sentry-sdk         | 1.15.0     | pypi_0                      | pypi    |   |
| setproctitle       | 1.3.2      | pypi_0                      | pypi    |   |
| setuptools         | 52.0.0     | py37h06a4308_0              |         |   |
| six                | 1.16.0     | pyhd3eb1b0_0                |         |   |
| smmap              | 4.0.0      | pypi_0                      | pypi    |   |
| sqlite             | 3.36.0     | hc218d9a_0                  |         |   |
| threadpoolctl      | 2.2.0      | pypi_0                      | pypi    |   |
| tk                 | 8.6.10     | hbc83047_0                  |         |   |
| tokenizers         | 0.10.3     | pypi_0                      | pypi    |   |
| toml               | 0.10.2     | pypi_0                      | pypi    |   |
| torchaudio         | 0.9.0      | py37                        | pytorch |   |
| torchvision        | 0.10.0     | py37_cu111                  | pytorch |   |
| tqdm               | 4.61.2     | pypi_0                      | pypi    |   |
| traitlets          | 5.0.5      | pypi_0                      | pypi    |   |
| transformers       | 4.9.1      | pypi_0                      | pypi    |   |
| typing-extensions  | 3.10.0.0   | hd3eb1b0_0                  |         |   |
| typing_extensions  | 3.10.0.0   | pyh06a4308_0                |         |   |
| urllib3            | 1.26.14    | pypi_0                      | pypi    |   |
| wandb              | 0.13.10    | pypi_0                      | pypi    |   |
| wcwidth            | 0.2.5      | pypi_0                      | pypi    |   |
| wheel              | 0.36.2     | pyhd3eb1b0_0                |         |   |
| wrapt              | 1.12.1     | pypi_0                      | pypi    |   |
| xz                 | 5.2.5      | h7b6447c_0                  |         |   |
| zipp               | 3.5.0      | pypi_0                      | pypi    |   |
| zlib               | 1.2.11     | h7b6447c_3                  |         |   |
| zstd               | 1.4.9      | haebb681_0                  |         |   |


[Bug]504 Server Error when loading dataset which was already cached

I am experiencing the same problem with the following environment:

* `datasets` version: 2.11.0
* Platform: `Linux 5.19.0-41-generic x86_64 GNU/Linux`
* Python version: `3.8.5`
* Huggingface_hub version: 0.13.3
* PyArrow version: `11.0.0`
* Pandas version: `1.5.3`

Trying to get some diagnostics, I got the following:

```python
>>> from huggingface_hub import scan_cache_dir
>>> sd = scan_cache_dir()
>>> sd
HFCacheInfo(size_on_disk=0, repos=frozenset(), warnings=[CorruptedCacheException('Repo path is not a directory: /home/myname/.cache/huggingface/hub/version_diffusers_cache.txt')])

```
However, that might also be because I had tried to manually specify the `cache_dir` and that resulted in trying to download the dataset again ... but into a folder one level higher up than it should have.

Note that my issue is with the `huggan/wikiart` dataset, so it is not a dataset-specific issue.

same problem with a private dataset repo, seems the huggingface hub server got some connection problem?

@SingL3 You can avoid this error by setting the [`HF_DATASETS_OFFLINE`](https://huggingface.co/docs/datasets/v2.12.0/en/loading#offline) env variable to 1. By default,  if an internet connection is available, we check whether the cache of a cached dataset is up-to-date.

@lucidBrot `datasets`' cache is still not aligned with `huggigface_hub`'s. We plan to align it eventually.

Today we had a big issue affecting the Hugging Face Hub, thus all the `504 Server Error: Gateway Time-out` errors.

It is fixed now and loading your datasets should work as expected.

Hi, @albertvillanova.
If there is a locally cached version of datasets or something cache using huggingface_hub, when a network problem(either client or server) occurs, is it a better way to fallback to use the current cached version rather than raise a exception and exit?

### Describe the bug

I have already cached the dataset using:
```
dataset = load_dataset("databricks/databricks-dolly-15k",
                        cache_dir="/mnt/data/llm/datasets/databricks-dolly-15k")
```
After that, I tried to load it again using the same machine, I got this error:
```
Traceback (most recent call last):
  File "/mnt/home/llm/pythia/train.py", line 16, in <module>
    dataset = load_dataset("databricks/databricks-dolly-15k",
  File "/mnt/data/conda/envs/pythia_ft/lib/python3.9/site-packages/datasets/load.py", line 1773, in load_dataset
    builder_instance = load_dataset_builder(
  File "/mnt/data/conda/envs/pythia_ft/lib/python3.9/site-packages/datasets/load.py", line 1502, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/mnt/data/conda/envs/pythia_ft/lib/python3.9/site-packages/datasets/load.py", line 1219, in dataset_module_factory
    raise e1 from None
  File "/mnt/data/conda/envs/pythia_ft/lib/python3.9/site-packages/datasets/load.py", line 1186, in dataset_module_factory
    raise e
  File "/mnt/data/conda/envs/pythia_ft/lib/python3.9/site-packages/datasets/load.py", line 1160, in dataset_module_factory
    dataset_info = hf_api.dataset_info(
  File "/mnt/data/conda/envs/pythia_ft/lib/python3.9/site-packages/huggingface_hub/utils/_validators.py", line 120, in _inner_fn
    return fn(*args, **kwargs)
  File "/mnt/data/conda/envs/pythia_ft/lib/python3.9/site-packages/huggingface_hub/hf_api.py", line 1667, in dataset_info
    hf_raise_for_status(r)
  File "/mnt/data/conda/envs/pythia_ft/lib/python3.9/site-packages/huggingface_hub/utils/_errors.py", line 301, in hf_raise_for_status
    raise HfHubHTTPError(str(e), response=response) from e
huggingface_hub.utils._errors.HfHubHTTPError: 504 Server Error: Gateway Time-out for url: https://huggingface.co/api/datasets/databricks/databricks-dolly-15k
```

### Steps to reproduce the bug

1. cache the databrick-dolly-15k dataset using load_dataset, setting a cache_dir
2. use load_dataset again, setting the same cache_dir

### Expected behavior

Dataset loaded succuessfully.

### Environment info

- `datasets` version: 2.12.0
- Platform: Linux-4.18.0-372.16.1.el8_6.x86_64-x86_64-with-glibc2.27
- Python version: 3.9.16
- Huggingface_hub version: 0.14.1
- PyArrow version: 11.0.0
- Pandas version: 1.5.3


Debug windows #2


(mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64'))

That is weird. I can't reproduce it again after reboot.
```python
In [2]: import platform

In [3]: platform.platform()
Out[3]: 'macOS-13.2-arm64-arm-64bit'

In [4]: from datasets import load_dataset
   ...:
   ...: jazzy = load_dataset("nomic-ai/gpt4all-j-prompt-generations", revision='v1.2-jazzy')
Found cached dataset parquet (/Users/sarit/.cache/huggingface/datasets/nomic-ai___parquet/nomic-ai--gpt4all-j-prompt-generations-a3b62015e2e52043/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 63.25it/s]
```

### Describe the bug

M2 MBP can't run
```python
from datasets import load_dataset

jazzy = load_dataset("nomic-ai/gpt4all-j-prompt-generations", revision='v1.2-jazzy')
```

### Steps to reproduce the bug

1. Use M2 MBP
2. Python 3.10.10 from pyenv
3. Run
```
from datasets import load_dataset

jazzy = load_dataset("nomic-ai/gpt4all-j-prompt-generations", revision='v1.2-jazzy')
```

### Expected behavior

Be able to run normally

### Environment info

```
from datasets import load_dataset

jazzy = load_dataset("nomic-ai/gpt4all-j-prompt-generations", revision='v1.2-jazzy')
```
OSX: 13.2
CPU: M2


Stream data concatenation issue

Hi!

You can call `map` as follows to avoid the error:
```python
augmented_dataset_cln = dataset_cln['train'].map(augment_dataset, features=dataset_cln['train'].features)
```

Hi!
I have run into the same problem with you. Could you please let me know how you solve it? Thanks!

### Describe the bug

I am not able to concatenate the augmentation of the stream data. I am using the latest version of dataset.

ValueError: The features can't be aligned because the key audio of features {'audio_id': Value(dtype='string',
id=None), 'audio': {'array': Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), 'path':
Value(dtype='null', id=None), 'sampling_rate': Value(dtype='int64', id=None)}, 'transcript': Value(dtype='string',
id=None)} has unexpected type - {'array': Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None),
'path': Value(dtype='null', id=None), 'sampling_rate': Value(dtype='int64', id=None)} (expected either
Audio(sampling_rate=16000, mono=True, decode=True, id=None) or Value("null").

### Steps to reproduce the bug

dataset = load_dataset("tobiolatunji/afrispeech-200", "all", streaming=True).shuffle(seed=42)
dataset_cln = dataset.remove_columns(['speaker_id', 'path', 'age_group', 'gender', 'accent', 'domain', 'country', 'duration'])
dataset_cln = dataset_cln.cast_column("audio", Audio(sampling_rate=16000))
from audiomentations import AddGaussianNoise,Compose,Gain,OneOf,PitchShift,PolarityInversion,TimeStretch

augmentation = Compose([
    AddGaussianNoise(min_amplitude=0.005, max_amplitude=0.015, p=0.2)
])

def augment_dataset(batch):
    audio = batch["audio"]
    audio["array"] = augmentation(audio["array"], sample_rate=audio["sampling_rate"])
    return batch

augmented_dataset_cln = dataset_cln['train'].map(augment_dataset)
dataset_cln['train'] = interleave_datasets([dataset_cln['train'], augmented_dataset_cln])
dataset_cln['train'] = dataset_cln['train'].shuffle(seed=42)

### Expected behavior

I should be able to merge as sampling rate is same.

### Environment info

import datasets
import transformers
import accelerate
print(datasets.__version__)
print(transformers.__version__)
print(torch.__version__)
print(evaluate.__version__)
print(accelerate.__version__)

2.12.0
4.28.1
2.0.0
0.4.0
0.18.0


load json dataset interrupt when dtype cast problem occured

Indeed the JSON dataset builder raises an error when it encounters an unexpected type.

There's an old PR open to add away to ignore such elements though, if it can help: https://github.com/huggingface/datasets/pull/2838

### Describe the bug

i have a json like this:
[
    {"id": 1, "name": 1},
    {"id": 2, "name": "Nan"},
    {"id": 3, "name": 3},
    ....
]
，which have several problematic rows data like row 2, then i load it with datasets.load_dataset('json', data_files=['xx.json'], split='train'), it will report like this:
Generating train split: 0 examples [00:00, ? examples/s]Failed to read file 'C:\Users\gawinjunwu\Downloads\test\data\a.json' with error <class 'pyarrow.lib.ArrowInvalid'>: Could not convert '2' with type str: tried to convert to int64
Traceback (most recent call last):
  File "D:\Python3.9\lib\site-packages\datasets\builder.py", line 1858, in _prepare_split_single
    for _, table in generator:
  File "D:\Python3.9\lib\site-packages\datasets\packaged_modules\json\json.py", line 146, in _generate_tables
    raise ValueError(f"Not able to read records in the JSON file at {file}.") from None
ValueError: Not able to read records in the JSON file at C:\Users\gawinjunwu\Downloads\test\data\a.json.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "c:\Users\gawinjunwu\Downloads\test\scripts\a.py", line 4, in <module>
    ds = load_dataset('json', data_dir='data', split='train')
  File "D:\Python3.9\lib\site-packages\datasets\load.py", line 1797, in load_dataset
    builder_instance.download_and_prepare(
  File "D:\Python3.9\lib\site-packages\datasets\builder.py", line 890, in download_and_prepare
    self._download_and_prepare(
  File "D:\Python3.9\lib\site-packages\datasets\builder.py", line 985, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "D:\Python3.9\lib\site-packages\datasets\builder.py", line 1746, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "D:\Python3.9\lib\site-packages\datasets\builder.py", line 1891, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset.
Could datasets skip those problematic data row?

### Steps to reproduce the bug

prepare a json file like this:
[
    {"id": 1, "name": 1},
    {"id": 2, "name": "Nan"},
    {"id": 3, "name": 3}
]
then use datasets.load_dataset('json', dir_files=['xxx.json']) to load the json file

### Expected behavior

skip the problematic data row and load row1 and row3

### Environment info

python3.9


Support working_dir in from_spark

_The documentation is not available anymore as the PR was closed or merged._

@lhoestq would you or another maintainer be able to review please? :)

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005771 / 0.011353 (-0.005582) | 0.004086 / 0.011008 (-0.006922) | 0.097170 / 0.038508 (0.058661) | 0.027464 / 0.023109 (0.004355) | 0.305425 / 0.275898 (0.029527) | 0.343869 / 0.323480 (0.020389) | 0.004899 / 0.007986 (-0.003087) | 0.003294 / 0.004328 (-0.001034) | 0.074710 / 0.004250 (0.070459) | 0.034982 / 0.037052 (-0.002070) | 0.306063 / 0.258489 (0.047574) | 0.343115 / 0.293841 (0.049274) | 0.025155 / 0.128546 (-0.103392) | 0.008429 / 0.075646 (-0.067217) | 0.318680 / 0.419271 (-0.100591) | 0.043304 / 0.043533 (-0.000229) | 0.306703 / 0.255139 (0.051564) | 0.335535 / 0.283200 (0.052335) | 0.087428 / 0.141683 (-0.054255) | 1.483769 / 1.452155 (0.031614) | 1.538753 / 1.492716 (0.046037) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.203313 / 0.018006 (0.185307) | 0.413864 / 0.000490 (0.413375) | 0.003186 / 0.000200 (0.002986) | 0.000068 / 0.000054 (0.000013) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022862 / 0.037411 (-0.014550) | 0.097306 / 0.014526 (0.082780) | 0.102823 / 0.176557 (-0.073733) | 0.162803 / 0.737135 (-0.574333) | 0.106311 / 0.296338 (-0.190028) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.451710 / 0.215209 (0.236501) | 4.508520 / 2.077655 (2.430865) | 2.181118 / 1.504120 (0.676998) | 1.977607 / 1.541195 (0.436412) | 2.008366 / 1.468490 (0.539876) | 0.565388 / 4.584777 (-4.019389) | 3.439318 / 3.745712 (-0.306394) | 1.747512 / 5.269862 (-3.522349) | 1.102124 / 4.565676 (-3.463553) | 0.069212 / 0.424275 (-0.355063) | 0.011926 / 0.007607 (0.004318) | 0.553414 / 0.226044 (0.327370) | 5.548959 / 2.268929 (3.280031) | 2.628769 / 55.444624 (-52.815856) | 2.301003 / 6.876477 (-4.575473) | 2.341744 / 2.142072 (0.199672) | 0.673092 / 4.805227 (-4.132135) | 0.137722 / 6.500664 (-6.362942) | 0.066909 / 0.075469 (-0.008560) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.196854 / 1.841788 (-0.644934) | 13.421776 / 8.074308 (5.347468) | 13.839760 / 10.191392 (3.648368) | 0.140557 / 0.680424 (-0.539867) | 0.016619 / 0.534201 (-0.517582) | 0.357985 / 0.579283 (-0.221298) | 0.387018 / 0.434364 (-0.047346) | 0.452798 / 0.540337 (-0.087540) | 0.542085 / 1.386936 (-0.844851) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005868 / 0.011353 (-0.005484) | 0.004103 / 0.011008 (-0.006905) | 0.076126 / 0.038508 (0.037618) | 0.027744 / 0.023109 (0.004635) | 0.357257 / 0.275898 (0.081359) | 0.387981 / 0.323480 (0.064501) | 0.004807 / 0.007986 (-0.003178) | 0.003337 / 0.004328 (-0.000991) | 0.075486 / 0.004250 (0.071236) | 0.035121 / 0.037052 (-0.001931) | 0.361385 / 0.258489 (0.102896) | 0.399346 / 0.293841 (0.105505) | 0.025263 / 0.128546 (-0.103284) | 0.008571 / 0.075646 (-0.067075) | 0.081815 / 0.419271 (-0.337457) | 0.041114 / 0.043533 (-0.002418) | 0.362840 / 0.255139 (0.107701) | 0.380926 / 0.283200 (0.097727) | 0.092728 / 0.141683 (-0.048955) | 1.517647 / 1.452155 (0.065492) | 1.534914 / 1.492716 (0.042198) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.199669 / 0.018006 (0.181663) | 0.399070 / 0.000490 (0.398580) | 0.002014 / 0.000200 (0.001814) | 0.000079 / 0.000054 (0.000024) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024541 / 0.037411 (-0.012870) | 0.099676 / 0.014526 (0.085151) | 0.106503 / 0.176557 (-0.070054) | 0.153755 / 0.737135 (-0.583380) | 0.108564 / 0.296338 (-0.187775) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.443842 / 0.215209 (0.228633) | 4.441158 / 2.077655 (2.363503) | 2.159496 / 1.504120 (0.655376) | 1.955358 / 1.541195 (0.414163) | 1.973864 / 1.468490 (0.505374) | 0.550467 / 4.584777 (-4.034310) | 3.381831 / 3.745712 (-0.363881) | 2.561192 / 5.269862 (-2.708670) | 1.361684 / 4.565676 (-3.203992) | 0.068140 / 0.424275 (-0.356135) | 0.012005 / 0.007607 (0.004398) | 0.551921 / 0.226044 (0.325877) | 5.503591 / 2.268929 (3.234662) | 2.591609 / 55.444624 (-52.853015) | 2.246681 / 6.876477 (-4.629796) | 2.290941 / 2.142072 (0.148868) | 0.655212 / 4.805227 (-4.150015) | 0.136013 / 6.500664 (-6.364651) | 0.066995 / 0.075469 (-0.008474) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.300438 / 1.841788 (-0.541350) | 13.866224 / 8.074308 (5.791916) | 13.932624 / 10.191392 (3.741232) | 0.144345 / 0.680424 (-0.536079) | 0.016623 / 0.534201 (-0.517578) | 0.357629 / 0.579283 (-0.221654) | 0.389759 / 0.434364 (-0.044605) | 0.417704 / 0.540337 (-0.122633) | 0.501358 / 1.386936 (-0.885578) |

![](https://cml.dev/watermark.png#89f775226321ba94e5bf4670a323c0fb44f5f65c "CML watermark")

Accept `working_dir` as an argument to `Dataset.from_spark`. Setting a non-NFS working directory for Spark workers to materialize to will improve write performance.


FileNotFound even though exists

Hi!

This would only work if `bigscience/xP3` was a no-code dataset, but it isn't (it has a Python builder script).

But this should work:
```python
load_dataset("json", data_files="https://huggingface.co/datasets/bigscience/xP3/resolve/main/ur/xp3_facebook_flores_spa_Latn-urd_Arab_devtest_ab-spa_Latn-urd_Arab.jsonl")
```

> I see, it's not compatible w/ regex right? e.g. `load_dataset("json", data_files="https://huggingface.co/datasets/bigscience/xP3/resolve/main/ur/*")`

It should work for patterns that "reference" the local filesystem, but to make this work with the Hub, we must implement https://github.com/huggingface/datasets/issues/5281 first.

In the meantime, you can fetch these glob files with `HfFileSystem` and pass them as a list to `load_dataset`:
```python
from datasets import load_dataset
from huggingface_hub import HfFileSystem, hf_hub_url # `HfFileSystem` requires the latest version of `huggingface_hub`

fs = HfFileSystem()
glob_files = fs.glob("datasets/bigscience/xP3/ur/*")
# convert fsspec URLs to HTTP URLs
resolved_paths = [fs.resolve_path(file) for file in glob_files]
data_files = [hf_hub_url(resolved_path.repo_id, resolved_path.path_in_repo, repo_type=resolved_path.repo_type) for resolved_path in resolved_paths]

ds = load_dataset("json", data_files=data_files)
```

### Describe the bug

I'm trying to download https://huggingface.co/datasets/bigscience/xP3/resolve/main/ur/xp3_facebook_flores_spa_Latn-urd_Arab_devtest_ab-spa_Latn-urd_Arab.jsonl which works fine in my webbrowser, but somehow not with datasets. Am I doing sth wrong?

```
Downloading builder script: 100%
2.82k/2.82k [00:00<00:00, 64.2kB/s]
Downloading readme: 100%
12.6k/12.6k [00:00<00:00, 585kB/s]
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
[<ipython-input-2-4b45446a91d5>](https://localhost:8080/#) in <cell line: 4>()
      2 lang = "ur"
      3 fname = "xp3_facebook_flores_spa_Latn-urd_Arab_devtest_ab-spa_Latn-urd_Arab.jsonl"
----> 4 dataset = load_dataset("bigscience/xP3", data_files=f"{lang}/{fname}")

6 frames
[/usr/local/lib/python3.10/dist-packages/datasets/data_files.py](https://localhost:8080/#) in _resolve_single_pattern_locally(base_path, pattern, allowed_extensions)
    291         if allowed_extensions is not None:
    292             error_msg += f" with any supported extension {list(allowed_extensions)}"
--> 293         raise FileNotFoundError(error_msg)
    294     return sorted(out)
    295

FileNotFoundError: Unable to find 'https://huggingface.co/datasets/bigscience/xP3/resolve/main/ur/xp3_facebook_flores_spa_Latn-urd_Arab_devtest_ab-spa_Latn-urd_Arab.jsonl' at /content/https:/huggingface.co/datasets/bigscience/xP3/resolve/main
```

### Steps to reproduce the bug

```
!pip install -q datasets
from datasets import load_dataset
lang = "ur"
fname = "xp3_facebook_flores_spa_Latn-urd_Arab_devtest_ab-spa_Latn-urd_Arab.jsonl"
dataset = load_dataset("bigscience/xP3", data_files=f"{lang}/{fname}")
```

### Expected behavior

Correctly downloads

### Environment info

latest versions