Better row group size in push_to_hub

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007489 / 0.011353 (-0.003864) | 0.004914 / 0.011008 (-0.006095) | 0.111626 / 0.038508 (0.073117) | 0.037920 / 0.023109 (0.014811) | 0.350571 / 0.275898 (0.074673) | 0.389667 / 0.323480 (0.066187) | 0.006309 / 0.007986 (-0.001676) | 0.005488 / 0.004328 (0.001160) | 0.083962 / 0.004250 (0.079712) | 0.050728 / 0.037052 (0.013675) | 0.360997 / 0.258489 (0.102508) | 0.392736 / 0.293841 (0.098895) | 0.031975 / 0.128546 (-0.096571) | 0.009941 / 0.075646 (-0.065705) | 0.379840 / 0.419271 (-0.039432) | 0.056522 / 0.043533 (0.012989) | 0.359379 / 0.255139 (0.104240) | 0.384487 / 0.283200 (0.101287) | 0.117523 / 0.141683 (-0.024160) | 1.683639 / 1.452155 (0.231485) | 1.791645 / 1.492716 (0.298929) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.236862 / 0.018006 (0.218856) | 0.481208 / 0.000490 (0.480719) | 0.007455 / 0.000200 (0.007255) | 0.000111 / 0.000054 (0.000057) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030854 / 0.037411 (-0.006557) | 0.126892 / 0.014526 (0.112367) | 0.139207 / 0.176557 (-0.037350) | 0.206447 / 0.737135 (-0.530689) | 0.143095 / 0.296338 (-0.153244) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.474677 / 0.215209 (0.259468) | 4.699534 / 2.077655 (2.621879) | 2.152102 / 1.504120 (0.647983) | 1.934815 / 1.541195 (0.393620) | 1.986448 / 1.468490 (0.517958) | 0.607184 / 4.584777 (-3.977593) | 4.480385 / 3.745712 (0.734673) | 2.074729 / 5.269862 (-3.195132) | 1.182383 / 4.565676 (-3.383294) | 0.075624 / 0.424275 (-0.348651) | 0.014046 / 0.007607 (0.006439) | 0.598859 / 0.226044 (0.372814) | 5.959551 / 2.268929 (3.690622) | 2.700851 / 55.444624 (-52.743773) | 2.303775 / 6.876477 (-4.572702) | 2.456441 / 2.142072 (0.314369) | 0.747185 / 4.805227 (-4.058042) | 0.165787 / 6.500664 (-6.334878) | 0.075817 / 0.075469 (0.000348) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.411859 / 1.841788 (-0.429928) | 17.375495 / 8.074308 (9.301187) | 15.187098 / 10.191392 (4.995706) | 0.169953 / 0.680424 (-0.510471) | 0.020204 / 0.534201 (-0.513997) | 0.461424 / 0.579283 (-0.117859) | 0.494443 / 0.434364 (0.060080) | 0.544583 / 0.540337 (0.004246) | 0.648231 / 1.386936 (-0.738705) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007785 / 0.011353 (-0.003568) | 0.005314 / 0.011008 (-0.005694) | 0.087273 / 0.038508 (0.048765) | 0.037810 / 0.023109 (0.014701) | 0.425473 / 0.275898 (0.149575) | 0.459976 / 0.323480 (0.136497) | 0.007270 / 0.007986 (-0.000716) | 0.004631 / 0.004328 (0.000303) | 0.087063 / 0.004250 (0.082812) | 0.052630 / 0.037052 (0.015578) | 0.432384 / 0.258489 (0.173895) | 0.500291 / 0.293841 (0.206450) | 0.033144 / 0.128546 (-0.095402) | 0.010101 / 0.075646 (-0.065545) | 0.096068 / 0.419271 (-0.323204) | 0.062750 / 0.043533 (0.019217) | 0.419308 / 0.255139 (0.164169) | 0.437099 / 0.283200 (0.153900) | 0.122289 / 0.141683 (-0.019394) | 1.737829 / 1.452155 (0.285674) | 1.851481 / 1.492716 (0.358765) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.014277 / 0.018006 (-0.003729) | 0.489835 / 0.000490 (0.489345) | 0.008423 / 0.000200 (0.008223) | 0.000188 / 0.000054 (0.000134) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032966 / 0.037411 (-0.004445) | 0.130069 / 0.014526 (0.115544) | 0.144372 / 0.176557 (-0.032185) | 0.200400 / 0.737135 (-0.536735) | 0.149384 / 0.296338 (-0.146954) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.511542 / 0.215209 (0.296333) | 5.093879 / 2.077655 (3.016225) | 2.572088 / 1.504120 (1.067968) | 2.339118 / 1.541195 (0.797923) | 2.441637 / 1.468490 (0.973147) | 0.614818 / 4.584777 (-3.969959) | 4.724441 / 3.745712 (0.978729) | 5.431978 / 5.269862 (0.162116) | 2.257794 / 4.565676 (-2.307883) | 0.078109 / 0.424275 (-0.346166) | 0.013821 / 0.007607 (0.006214) | 0.639232 / 0.226044 (0.413188) | 6.424623 / 2.268929 (4.155694) | 3.163018 / 55.444624 (-52.281606) | 2.756786 / 6.876477 (-4.119690) | 2.808655 / 2.142072 (0.666583) | 0.745843 / 4.805227 (-4.059385) | 0.165562 / 6.500664 (-6.335102) | 0.076610 / 0.075469 (0.001141) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.738630 / 1.841788 (-0.103158) | 18.073573 / 8.074308 (9.999265) | 16.482820 / 10.191392 (6.291428) | 0.213233 / 0.680424 (-0.467191) | 0.022839 / 0.534201 (-0.511362) | 0.487043 / 0.579283 (-0.092240) | 0.512518 / 0.434364 (0.078154) | 0.549365 / 0.540337 (0.009028) | 0.656612 / 1.386936 (-0.730324) |

![](https://cml.dev/watermark.png#288e92b03bd4ec91c10c8a529b32631cfaba9fb7 "CML watermark")

Good idea!

I was wondering: if we want to optimize the balance between the size of downloading a row group, and the number of rows in the group, would it make sense to compute the row group size by checking the average size of the rows?

eg. 32x32 images could have a larger row group size than full HD images, no? Relying on the size would even remove the need to check the column types.

(in this proposal, we could use the computed row group size, eg 837, or use the nearest row group size in a list of values: 10, 100, 1000, 10000)

Probably, but I would go for a simpler solution first :p

Sure! I wanted to understand if the idea made sense or not, but it's not for this PR.

I think it will be more useful for people who use the viewer and won't impact sequential io that much.

DuckDB [paragraph](https://duckdb.org/docs/data/parquet/tips.html#selecting-a-row_group_size) that explains how to choose the `row_group_size`. Our default shard size is 500 MB in `push_to_hub`, so, ideally, we should aim for 64 MB row groups (and make this part configurable for power users 🙂).

So, before merging this PR, let's add a TODO or open an issue as a reminder that this can be improved.

I moved the config values, improved the features check and mentioned the improvements we could do in the docstring :)

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006211 / 0.011353 (-0.005141) | 0.004244 / 0.011008 (-0.006764) | 0.097941 / 0.038508 (0.059433) | 0.028564 / 0.023109 (0.005455) | 0.299651 / 0.275898 (0.023753) | 0.340694 / 0.323480 (0.017214) | 0.005161 / 0.007986 (-0.002824) | 0.004764 / 0.004328 (0.000435) | 0.075505 / 0.004250 (0.071255) | 0.039656 / 0.037052 (0.002603) | 0.309242 / 0.258489 (0.050753) | 0.350783 / 0.293841 (0.056942) | 0.025145 / 0.128546 (-0.103401) | 0.008498 / 0.075646 (-0.067148) | 0.317657 / 0.419271 (-0.101615) | 0.043926 / 0.043533 (0.000394) | 0.305915 / 0.255139 (0.050776) | 0.331630 / 0.283200 (0.048430) | 0.088564 / 0.141683 (-0.053119) | 1.533175 / 1.452155 (0.081021) | 1.581017 / 1.492716 (0.088301) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.206032 / 0.018006 (0.188025) | 0.433446 / 0.000490 (0.432956) | 0.003955 / 0.000200 (0.003755) | 0.000095 / 0.000054 (0.000040) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023468 / 0.037411 (-0.013943) | 0.103292 / 0.014526 (0.088766) | 0.107234 / 0.176557 (-0.069322) | 0.168525 / 0.737135 (-0.568610) | 0.113218 / 0.296338 (-0.183120) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.431085 / 0.215209 (0.215875) | 4.302082 / 2.077655 (2.224427) | 2.068290 / 1.504120 (0.564171) | 1.850718 / 1.541195 (0.309523) | 1.964261 / 1.468490 (0.495771) | 0.547562 / 4.584777 (-4.037215) | 3.410739 / 3.745712 (-0.334974) | 1.779640 / 5.269862 (-3.490221) | 1.005466 / 4.565676 (-3.560210) | 0.066250 / 0.424275 (-0.358025) | 0.011877 / 0.007607 (0.004270) | 0.525185 / 0.226044 (0.299141) | 5.234786 / 2.268929 (2.965857) | 2.398045 / 55.444624 (-53.046580) | 2.073020 / 6.876477 (-4.803457) | 2.210753 / 2.142072 (0.068680) | 0.654897 / 4.805227 (-4.150331) | 0.134639 / 6.500664 (-6.366025) | 0.067050 / 0.075469 (-0.008419) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.180210 / 1.841788 (-0.661577) | 13.613091 / 8.074308 (5.538783) | 13.441837 / 10.191392 (3.250445) | 0.146048 / 0.680424 (-0.534376) | 0.016505 / 0.534201 (-0.517696) | 0.363210 / 0.579283 (-0.216073) | 0.405484 / 0.434364 (-0.028880) | 0.428712 / 0.540337 (-0.111625) | 0.522300 / 1.386936 (-0.864636) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006147 / 0.011353 (-0.005206) | 0.004161 / 0.011008 (-0.006847) | 0.075861 / 0.038508 (0.037353) | 0.027948 / 0.023109 (0.004839) | 0.362466 / 0.275898 (0.086568) | 0.398227 / 0.323480 (0.074747) | 0.005014 / 0.007986 (-0.002972) | 0.004772 / 0.004328 (0.000444) | 0.075674 / 0.004250 (0.071423) | 0.039158 / 0.037052 (0.002106) | 0.363567 / 0.258489 (0.105078) | 0.410378 / 0.293841 (0.116537) | 0.025510 / 0.128546 (-0.103036) | 0.008528 / 0.075646 (-0.067118) | 0.081803 / 0.419271 (-0.337468) | 0.040954 / 0.043533 (-0.002579) | 0.358492 / 0.255139 (0.103353) | 0.381345 / 0.283200 (0.098145) | 0.092347 / 0.141683 (-0.049336) | 1.567695 / 1.452155 (0.115540) | 1.668412 / 1.492716 (0.175696) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.203367 / 0.018006 (0.185360) | 0.424642 / 0.000490 (0.424152) | 0.002451 / 0.000200 (0.002251) | 0.000071 / 0.000054 (0.000017) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026129 / 0.037411 (-0.011282) | 0.102564 / 0.014526 (0.088039) | 0.110583 / 0.176557 (-0.065973) | 0.164332 / 0.737135 (-0.572804) | 0.115706 / 0.296338 (-0.180632) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.468925 / 0.215209 (0.253716) | 4.657266 / 2.077655 (2.579612) | 2.423280 / 1.504120 (0.919160) | 2.236284 / 1.541195 (0.695089) | 2.323019 / 1.468490 (0.854529) | 0.548120 / 4.584777 (-4.036657) | 3.455602 / 3.745712 (-0.290110) | 1.730421 / 5.269862 (-3.539441) | 1.006089 / 4.565676 (-3.559588) | 0.067478 / 0.424275 (-0.356797) | 0.011465 / 0.007607 (0.003857) | 0.574235 / 0.226044 (0.348190) | 5.744404 / 2.268929 (3.475475) | 2.882225 / 55.444624 (-52.562400) | 2.618246 / 6.876477 (-4.258231) | 2.642920 / 2.142072 (0.500847) | 0.661441 / 4.805227 (-4.143787) | 0.137358 / 6.500664 (-6.363306) | 0.070372 / 0.075469 (-0.005097) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.333815 / 1.841788 (-0.507973) | 14.689667 / 8.074308 (6.615359) | 14.362294 / 10.191392 (4.170902) | 0.152011 / 0.680424 (-0.528413) | 0.016869 / 0.534201 (-0.517332) | 0.370433 / 0.579283 (-0.208851) | 0.399642 / 0.434364 (-0.034722) | 0.433759 / 0.540337 (-0.106578) | 0.525443 / 1.386936 (-0.861493) |

![](https://cml.dev/watermark.png#09e9f9a88edd9055b5c540e3d83b5a11d48f8ba8 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006564 / 0.011353 (-0.004789) | 0.004350 / 0.011008 (-0.006658) | 0.096277 / 0.038508 (0.057769) | 0.032956 / 0.023109 (0.009847) | 0.303675 / 0.275898 (0.027777) | 0.336384 / 0.323480 (0.012904) | 0.005789 / 0.007986 (-0.002197) | 0.003957 / 0.004328 (-0.000371) | 0.073990 / 0.004250 (0.069740) | 0.050974 / 0.037052 (0.013922) | 0.321754 / 0.258489 (0.063265) | 0.349489 / 0.293841 (0.055648) | 0.031138 / 0.128546 (-0.097409) | 0.009000 / 0.075646 (-0.066646) | 0.325445 / 0.419271 (-0.093826) | 0.070173 / 0.043533 (0.026640) | 0.304706 / 0.255139 (0.049567) | 0.321803 / 0.283200 (0.038603) | 0.109405 / 0.141683 (-0.032278) | 1.489812 / 1.452155 (0.037657) | 1.577729 / 1.492716 (0.085013) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.287187 / 0.018006 (0.269181) | 0.527625 / 0.000490 (0.527135) | 0.006533 / 0.000200 (0.006333) | 0.000090 / 0.000054 (0.000036) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026659 / 0.037411 (-0.010752) | 0.106236 / 0.014526 (0.091710) | 0.118615 / 0.176557 (-0.057941) | 0.173156 / 0.737135 (-0.563979) | 0.122883 / 0.296338 (-0.173456) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.407189 / 0.215209 (0.191980) | 4.055732 / 2.077655 (1.978078) | 1.865594 / 1.504120 (0.361474) | 1.664325 / 1.541195 (0.123130) | 1.668961 / 1.468490 (0.200471) | 0.521207 / 4.584777 (-4.063570) | 3.740424 / 3.745712 (-0.005288) | 3.431973 / 5.269862 (-1.837889) | 1.636669 / 4.565676 (-2.929008) | 0.065271 / 0.424275 (-0.359005) | 0.012151 / 0.007607 (0.004544) | 0.514233 / 0.226044 (0.288189) | 5.110150 / 2.268929 (2.841222) | 2.264340 / 55.444624 (-53.180284) | 1.940428 / 6.876477 (-4.936049) | 2.042286 / 2.142072 (-0.099787) | 0.639200 / 4.805227 (-4.166028) | 0.139537 / 6.500664 (-6.361127) | 0.063195 / 0.075469 (-0.012274) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.179501 / 1.841788 (-0.662286) | 14.600133 / 8.074308 (6.525825) | 14.902137 / 10.191392 (4.710745) | 0.144509 / 0.680424 (-0.535915) | 0.017449 / 0.534201 (-0.516752) | 0.393135 / 0.579283 (-0.186148) | 0.413103 / 0.434364 (-0.021261) | 0.459897 / 0.540337 (-0.080440) | 0.552602 / 1.386936 (-0.834334) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006891 / 0.011353 (-0.004462) | 0.004633 / 0.011008 (-0.006375) | 0.073093 / 0.038508 (0.034585) | 0.032509 / 0.023109 (0.009399) | 0.348332 / 0.275898 (0.072434) | 0.381920 / 0.323480 (0.058440) | 0.005978 / 0.007986 (-0.002007) | 0.005360 / 0.004328 (0.001032) | 0.074307 / 0.004250 (0.070056) | 0.049668 / 0.037052 (0.012615) | 0.354713 / 0.258489 (0.096224) | 0.398521 / 0.293841 (0.104681) | 0.032013 / 0.128546 (-0.096534) | 0.008890 / 0.075646 (-0.066756) | 0.080013 / 0.419271 (-0.339259) | 0.051820 / 0.043533 (0.008288) | 0.349730 / 0.255139 (0.094591) | 0.369267 / 0.283200 (0.086067) | 0.103874 / 0.141683 (-0.037809) | 1.484148 / 1.452155 (0.031993) | 1.573927 / 1.492716 (0.081211) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.009699 / 0.018006 (-0.008307) | 0.511176 / 0.000490 (0.510686) | 0.002938 / 0.000200 (0.002738) | 0.000109 / 0.000054 (0.000054) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027847 / 0.037411 (-0.009564) | 0.111565 / 0.014526 (0.097039) | 0.120625 / 0.176557 (-0.055932) | 0.172130 / 0.737135 (-0.565006) | 0.125949 / 0.296338 (-0.170389) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.430634 / 0.215209 (0.215424) | 4.315377 / 2.077655 (2.237722) | 2.070764 / 1.504120 (0.566644) | 1.881962 / 1.541195 (0.340767) | 1.904053 / 1.468490 (0.435563) | 0.524973 / 4.584777 (-4.059804) | 3.718359 / 3.745712 (-0.027353) | 3.415344 / 5.269862 (-1.854518) | 1.224568 / 4.565676 (-3.341108) | 0.065593 / 0.424275 (-0.358682) | 0.011643 / 0.007607 (0.004036) | 0.537050 / 0.226044 (0.311006) | 5.352155 / 2.268929 (3.083226) | 2.557361 / 55.444624 (-52.887263) | 2.217770 / 6.876477 (-4.658707) | 2.194975 / 2.142072 (0.052902) | 0.635142 / 4.805227 (-4.170085) | 0.140642 / 6.500664 (-6.360022) | 0.064690 / 0.075469 (-0.010779) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.266125 / 1.841788 (-0.575663) | 14.836413 / 8.074308 (6.762105) | 14.446870 / 10.191392 (4.255478) | 0.191545 / 0.680424 (-0.488878) | 0.017433 / 0.534201 (-0.516768) | 0.392296 / 0.579283 (-0.186987) | 0.420698 / 0.434364 (-0.013666) | 0.463225 / 0.540337 (-0.077112) | 0.556127 / 1.386936 (-0.830809) |

![](https://cml.dev/watermark.png#7fcbe5b1575c8d162b65b9397b3dfda995a4e048 "CML watermark")

This is a very simple change that improves `to_parquet` to use a more reasonable row group size for image and audio datasets.

This is especially useful for `push_to_hub` and will provide a better experience with the dataset viewer on HF


Modify levels of some logging messages

I've addressed this as part of #6019, so feel free to close this PR.

Some warning messages didn't quite sound like warnings so I modified their logging levels to info.


Fix `to_numpy` when None values in the sequence

I just added the same test with dynamic shape

_The documentation is not available anymore as the PR was closed or merged._

Awesome ! I'm merging now if you don't mind :)
We should probably give you permissions to merge your own PRs when you have an approval

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009980 / 0.011353 (-0.001373) | 0.005709 / 0.011008 (-0.005300) | 0.132185 / 0.038508 (0.093677) | 0.039299 / 0.023109 (0.016190) | 0.400168 / 0.275898 (0.124270) | 0.470582 / 0.323480 (0.147102) | 0.007753 / 0.007986 (-0.000233) | 0.005196 / 0.004328 (0.000868) | 0.093698 / 0.004250 (0.089448) | 0.052631 / 0.037052 (0.015579) | 0.430347 / 0.258489 (0.171858) | 0.460162 / 0.293841 (0.166321) | 0.057511 / 0.128546 (-0.071035) | 0.013944 / 0.075646 (-0.061702) | 0.459008 / 0.419271 (0.039737) | 0.075532 / 0.043533 (0.031999) | 0.405165 / 0.255139 (0.150026) | 0.456142 / 0.283200 (0.172942) | 0.117309 / 0.141683 (-0.024374) | 1.945787 / 1.452155 (0.493633) | 2.067162 / 1.492716 (0.574446) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.285755 / 0.018006 (0.267749) | 0.619965 / 0.000490 (0.619476) | 0.005071 / 0.000200 (0.004871) | 0.000114 / 0.000054 (0.000059) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031112 / 0.037411 (-0.006299) | 0.128514 / 0.014526 (0.113988) | 0.137161 / 0.176557 (-0.039396) | 0.211363 / 0.737135 (-0.525772) | 0.151045 / 0.296338 (-0.145293) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.609361 / 0.215209 (0.394152) | 6.124844 / 2.077655 (4.047189) | 2.440757 / 1.504120 (0.936637) | 2.034495 / 1.541195 (0.493300) | 2.047192 / 1.468490 (0.578702) | 0.883171 / 4.584777 (-3.701606) | 5.470552 / 3.745712 (1.724840) | 4.401696 / 5.269862 (-0.868165) | 2.378674 / 4.565676 (-2.187003) | 0.108065 / 0.424275 (-0.316210) | 0.013239 / 0.007607 (0.005632) | 0.830957 / 0.226044 (0.604913) | 8.090659 / 2.268929 (5.821731) | 3.289203 / 55.444624 (-52.155422) | 2.500777 / 6.876477 (-4.375700) | 2.561440 / 2.142072 (0.419367) | 1.064893 / 4.805227 (-3.740334) | 0.220486 / 6.500664 (-6.280178) | 0.079507 / 0.075469 (0.004038) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.544334 / 1.841788 (-0.297454) | 17.878997 / 8.074308 (9.804689) | 18.952191 / 10.191392 (8.760799) | 0.245166 / 0.680424 (-0.435258) | 0.028022 / 0.534201 (-0.506179) | 0.517828 / 0.579283 (-0.061455) | 0.618988 / 0.434364 (0.184624) | 0.589742 / 0.540337 (0.049405) | 0.670902 / 1.386936 (-0.716034) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009616 / 0.011353 (-0.001737) | 0.006098 / 0.011008 (-0.004911) | 0.100301 / 0.038508 (0.061793) | 0.037792 / 0.023109 (0.014683) | 0.484667 / 0.275898 (0.208769) | 0.519286 / 0.323480 (0.195806) | 0.007427 / 0.007986 (-0.000558) | 0.007172 / 0.004328 (0.002844) | 0.104429 / 0.004250 (0.100179) | 0.056567 / 0.037052 (0.019515) | 0.502641 / 0.258489 (0.244152) | 0.549629 / 0.293841 (0.255788) | 0.049574 / 0.128546 (-0.078972) | 0.015223 / 0.075646 (-0.060424) | 0.113947 / 0.419271 (-0.305324) | 0.064585 / 0.043533 (0.021053) | 0.512962 / 0.255139 (0.257823) | 0.507218 / 0.283200 (0.224019) | 0.122194 / 0.141683 (-0.019488) | 1.927821 / 1.452155 (0.475667) | 2.051161 / 1.492716 (0.558445) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.291350 / 0.018006 (0.273344) | 0.588099 / 0.000490 (0.587610) | 0.001368 / 0.000200 (0.001168) | 0.000153 / 0.000054 (0.000099) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030604 / 0.037411 (-0.006807) | 0.126810 / 0.014526 (0.112285) | 0.139309 / 0.176557 (-0.037248) | 0.208030 / 0.737135 (-0.529105) | 0.138985 / 0.296338 (-0.157353) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.681254 / 0.215209 (0.466045) | 6.753856 / 2.077655 (4.676201) | 2.780704 / 1.504120 (1.276585) | 2.475205 / 1.541195 (0.934010) | 2.486784 / 1.468490 (1.018294) | 0.879223 / 4.584777 (-3.705554) | 5.662294 / 3.745712 (1.916582) | 2.698705 / 5.269862 (-2.571156) | 1.660620 / 4.565676 (-2.905057) | 0.112218 / 0.424275 (-0.312057) | 0.014211 / 0.007607 (0.006604) | 0.796957 / 0.226044 (0.570913) | 8.180897 / 2.268929 (5.911969) | 3.540419 / 55.444624 (-51.904205) | 2.899467 / 6.876477 (-3.977010) | 2.870306 / 2.142072 (0.728233) | 1.069537 / 4.805227 (-3.735690) | 0.211281 / 6.500664 (-6.289383) | 0.078898 / 0.075469 (0.003429) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.666790 / 1.841788 (-0.174998) | 18.302127 / 8.074308 (10.227819) | 21.317546 / 10.191392 (11.126153) | 0.242795 / 0.680424 (-0.437629) | 0.026754 / 0.534201 (-0.507447) | 0.493375 / 0.579283 (-0.085908) | 0.605400 / 0.434364 (0.171036) | 0.586888 / 0.540337 (0.046550) | 0.722809 / 1.386936 (-0.664127) |

![](https://cml.dev/watermark.png#ce2328e7b1d62998b22510492530af55d4493b73 "CML watermark")

Closes #5927
I've realized that the error was overlooked during testing due to the presence of only one None value in the sequence.
Unfortunately, it was the only case where the function works as expected. When the sequence contained more than one None value, the function failed. Consequently, I've updated the tests to include sequences with multiple None values.


[doc build] Use secrets

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008499 / 0.011353 (-0.002854) | 0.006155 / 0.011008 (-0.004853) | 0.124032 / 0.038508 (0.085524) | 0.037337 / 0.023109 (0.014228) | 0.389274 / 0.275898 (0.113376) | 0.427736 / 0.323480 (0.104257) | 0.006929 / 0.007986 (-0.001057) | 0.005017 / 0.004328 (0.000689) | 0.096356 / 0.004250 (0.092105) | 0.055694 / 0.037052 (0.018642) | 0.391417 / 0.258489 (0.132928) | 0.448098 / 0.293841 (0.154257) | 0.042442 / 0.128546 (-0.086105) | 0.013456 / 0.075646 (-0.062190) | 0.423502 / 0.419271 (0.004230) | 0.062919 / 0.043533 (0.019386) | 0.384317 / 0.255139 (0.129178) | 0.410851 / 0.283200 (0.127652) | 0.112807 / 0.141683 (-0.028875) | 1.746050 / 1.452155 (0.293895) | 1.977974 / 1.492716 (0.485257) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.306382 / 0.018006 (0.288375) | 0.620310 / 0.000490 (0.619820) | 0.009309 / 0.000200 (0.009109) | 0.000106 / 0.000054 (0.000052) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026900 / 0.037411 (-0.010511) | 0.140125 / 0.014526 (0.125599) | 0.136295 / 0.176557 (-0.040261) | 0.207721 / 0.737135 (-0.529414) | 0.146328 / 0.296338 (-0.150011) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.616712 / 0.215209 (0.401503) | 6.237820 / 2.077655 (4.160166) | 2.503809 / 1.504120 (0.999689) | 2.129739 / 1.541195 (0.588544) | 2.160768 / 1.468490 (0.692277) | 0.971273 / 4.584777 (-3.613504) | 5.687161 / 3.745712 (1.941449) | 2.738148 / 5.269862 (-2.531713) | 1.692695 / 4.565676 (-2.872981) | 0.113701 / 0.424275 (-0.310574) | 0.014809 / 0.007607 (0.007202) | 0.774795 / 0.226044 (0.548750) | 7.660012 / 2.268929 (5.391083) | 3.253036 / 55.444624 (-52.191588) | 2.607498 / 6.876477 (-4.268979) | 2.681678 / 2.142072 (0.539606) | 1.095275 / 4.805227 (-3.709952) | 0.239078 / 6.500664 (-6.261586) | 0.081034 / 0.075469 (0.005565) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.574547 / 1.841788 (-0.267240) | 18.323566 / 8.074308 (10.249258) | 19.274482 / 10.191392 (9.083090) | 0.210275 / 0.680424 (-0.470149) | 0.031843 / 0.534201 (-0.502358) | 0.514843 / 0.579283 (-0.064440) | 0.633782 / 0.434364 (0.199418) | 0.588569 / 0.540337 (0.048232) | 0.721401 / 1.386936 (-0.665535) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008866 / 0.011353 (-0.002487) | 0.006460 / 0.011008 (-0.004548) | 0.121337 / 0.038508 (0.082829) | 0.033896 / 0.023109 (0.010786) | 0.455702 / 0.275898 (0.179804) | 0.509685 / 0.323480 (0.186205) | 0.007650 / 0.007986 (-0.000336) | 0.005578 / 0.004328 (0.001250) | 0.098505 / 0.004250 (0.094255) | 0.056122 / 0.037052 (0.019069) | 0.478483 / 0.258489 (0.219994) | 0.560008 / 0.293841 (0.266167) | 0.044926 / 0.128546 (-0.083620) | 0.014562 / 0.075646 (-0.061085) | 0.115027 / 0.419271 (-0.304244) | 0.066494 / 0.043533 (0.022961) | 0.463434 / 0.255139 (0.208296) | 0.513856 / 0.283200 (0.230656) | 0.126436 / 0.141683 (-0.015247) | 1.874729 / 1.452155 (0.422575) | 1.925080 / 1.492716 (0.432364) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.012672 / 0.018006 (-0.005334) | 0.615797 / 0.000490 (0.615307) | 0.001606 / 0.000200 (0.001406) | 0.000118 / 0.000054 (0.000064) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031104 / 0.037411 (-0.006307) | 0.130107 / 0.014526 (0.115581) | 0.140587 / 0.176557 (-0.035970) | 0.205081 / 0.737135 (-0.532054) | 0.144068 / 0.296338 (-0.152270) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.646549 / 0.215209 (0.431340) | 6.403962 / 2.077655 (4.326307) | 2.812594 / 1.504120 (1.308474) | 2.478480 / 1.541195 (0.937285) | 2.552385 / 1.468490 (1.083895) | 0.991987 / 4.584777 (-3.592790) | 5.777917 / 3.745712 (2.032205) | 5.697830 / 5.269862 (0.427969) | 2.370583 / 4.565676 (-2.195094) | 0.109905 / 0.424275 (-0.314370) | 0.013801 / 0.007607 (0.006193) | 0.799932 / 0.226044 (0.573888) | 8.155672 / 2.268929 (5.886743) | 3.711662 / 55.444624 (-51.732963) | 3.042164 / 6.876477 (-3.834312) | 3.073549 / 2.142072 (0.931477) | 1.137515 / 4.805227 (-3.667712) | 0.231266 / 6.500664 (-6.269398) | 0.080893 / 0.075469 (0.005424) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.669210 / 1.841788 (-0.172577) | 18.747144 / 8.074308 (10.672836) | 21.084589 / 10.191392 (10.893197) | 0.241379 / 0.680424 (-0.439045) | 0.029473 / 0.534201 (-0.504728) | 0.524605 / 0.579283 (-0.054678) | 0.622852 / 0.434364 (0.188488) | 0.604941 / 0.540337 (0.064604) | 0.715978 / 1.386936 (-0.670958) |

![](https://cml.dev/watermark.png#142484a60b1330359d7713e906fc9e5e30aa9f64 "CML watermark")

Cool ! what about `.github/workflows/build_pr_documentation.yml` and `.github/workflows/delete_doc_comment.yml` ?

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005973 / 0.011353 (-0.005380) | 0.004389 / 0.011008 (-0.006620) | 0.096076 / 0.038508 (0.057568) | 0.031569 / 0.023109 (0.008460) | 0.328300 / 0.275898 (0.052402) | 0.359356 / 0.323480 (0.035876) | 0.005378 / 0.007986 (-0.002607) | 0.003703 / 0.004328 (-0.000625) | 0.075251 / 0.004250 (0.071000) | 0.042340 / 0.037052 (0.005287) | 0.346103 / 0.258489 (0.087614) | 0.379896 / 0.293841 (0.086055) | 0.027493 / 0.128546 (-0.101053) | 0.009033 / 0.075646 (-0.066613) | 0.327829 / 0.419271 (-0.091442) | 0.064074 / 0.043533 (0.020541) | 0.337703 / 0.255139 (0.082564) | 0.355335 / 0.283200 (0.072136) | 0.101179 / 0.141683 (-0.040504) | 1.471738 / 1.452155 (0.019584) | 1.539031 / 1.492716 (0.046315) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.194097 / 0.018006 (0.176091) | 0.434190 / 0.000490 (0.433701) | 0.005730 / 0.000200 (0.005530) | 0.000088 / 0.000054 (0.000033) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025634 / 0.037411 (-0.011778) | 0.105080 / 0.014526 (0.090555) | 0.116508 / 0.176557 (-0.060049) | 0.173867 / 0.737135 (-0.563269) | 0.117749 / 0.296338 (-0.178590) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.401566 / 0.215209 (0.186357) | 4.003558 / 2.077655 (1.925903) | 1.802756 / 1.504120 (0.298636) | 1.604222 / 1.541195 (0.063027) | 1.656617 / 1.468490 (0.188127) | 0.523385 / 4.584777 (-4.061392) | 3.744292 / 3.745712 (-0.001420) | 1.794295 / 5.269862 (-3.475567) | 1.044690 / 4.565676 (-3.520987) | 0.064992 / 0.424275 (-0.359284) | 0.011542 / 0.007607 (0.003935) | 0.507830 / 0.226044 (0.281785) | 5.061574 / 2.268929 (2.792645) | 2.252896 / 55.444624 (-53.191729) | 1.912551 / 6.876477 (-4.963926) | 2.073510 / 2.142072 (-0.068562) | 0.642148 / 4.805227 (-4.163079) | 0.140151 / 6.500664 (-6.360513) | 0.062623 / 0.075469 (-0.012846) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.180367 / 1.841788 (-0.661421) | 14.263475 / 8.074308 (6.189167) | 12.917251 / 10.191392 (2.725859) | 0.143815 / 0.680424 (-0.536608) | 0.017286 / 0.534201 (-0.516915) | 0.388411 / 0.579283 (-0.190872) | 0.430512 / 0.434364 (-0.003851) | 0.466595 / 0.540337 (-0.073742) | 0.564545 / 1.386936 (-0.822391) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006059 / 0.011353 (-0.005294) | 0.004419 / 0.011008 (-0.006590) | 0.074206 / 0.038508 (0.035697) | 0.031180 / 0.023109 (0.008071) | 0.380031 / 0.275898 (0.104133) | 0.410373 / 0.323480 (0.086893) | 0.005397 / 0.007986 (-0.002589) | 0.003952 / 0.004328 (-0.000376) | 0.074426 / 0.004250 (0.070176) | 0.046256 / 0.037052 (0.009203) | 0.385543 / 0.258489 (0.127054) | 0.430724 / 0.293841 (0.136883) | 0.028052 / 0.128546 (-0.100494) | 0.008810 / 0.075646 (-0.066836) | 0.080749 / 0.419271 (-0.338522) | 0.046746 / 0.043533 (0.003214) | 0.380325 / 0.255139 (0.125186) | 0.398901 / 0.283200 (0.115701) | 0.099607 / 0.141683 (-0.042076) | 1.433343 / 1.452155 (-0.018812) | 1.520447 / 1.492716 (0.027730) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.202232 / 0.018006 (0.184225) | 0.431342 / 0.000490 (0.430852) | 0.001020 / 0.000200 (0.000820) | 0.000089 / 0.000054 (0.000035) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028762 / 0.037411 (-0.008649) | 0.111777 / 0.014526 (0.097251) | 0.119283 / 0.176557 (-0.057273) | 0.168151 / 0.737135 (-0.568985) | 0.126093 / 0.296338 (-0.170245) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.442689 / 0.215209 (0.227480) | 4.369202 / 2.077655 (2.291547) | 2.167703 / 1.504120 (0.663583) | 1.960580 / 1.541195 (0.419385) | 2.001459 / 1.468490 (0.532969) | 0.527169 / 4.584777 (-4.057608) | 3.738987 / 3.745712 (-0.006726) | 1.819002 / 5.269862 (-3.450860) | 1.082786 / 4.565676 (-3.482891) | 0.066209 / 0.424275 (-0.358066) | 0.011549 / 0.007607 (0.003942) | 0.545959 / 0.226044 (0.319915) | 5.466655 / 2.268929 (3.197727) | 2.671448 / 55.444624 (-52.773176) | 2.340968 / 6.876477 (-4.535509) | 2.358805 / 2.142072 (0.216733) | 0.649456 / 4.805227 (-4.155771) | 0.142009 / 6.500664 (-6.358655) | 0.064199 / 0.075469 (-0.011270) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.259819 / 1.841788 (-0.581969) | 14.456988 / 8.074308 (6.382680) | 14.478982 / 10.191392 (4.287590) | 0.163156 / 0.680424 (-0.517268) | 0.017090 / 0.534201 (-0.517111) | 0.391339 / 0.579283 (-0.187944) | 0.422021 / 0.434364 (-0.012343) | 0.465340 / 0.540337 (-0.074997) | 0.564517 / 1.386936 (-0.822419) |

![](https://cml.dev/watermark.png#97358c88f996a65f49923ec215358044e4146a95 "CML watermark")

> .github/workflows/delete_doc_comment.yml

is already updated https://github.com/huggingface/datasets/pull/5932/files

> .github/workflows/build_pr_documentation.yml

indeed no changes are needed

Companion pr to https://github.com/huggingface/doc-builder/pull/379


`datasets.map` not reusing cached copy by default

This can happen when a map transform cannot be hashed deterministically (e.g., an object referenced by the transform changes its state after the first call - an issue with fast tokenizers). The solution is to provide `cache_file_name` in the `map` call to check this file for the cached result instead of relying on the default caching mechanism.

### Describe the bug

When I load the dataset from local directory, it's cached copy is picked up after first time. However, for `map` operation, the operation is applied again and cached copy is not picked up. Is there any way to pick cached copy instead of processing it again? The only solution I could think of was to use `save_to_disk` after my last transform and then use that in my DataLoader pipeline. Are there any other solutions for the same?

One more thing, my dataset is occupying 6GB storage memory after I use `map`, is there any way I can reduce that memory usage?

### Steps to reproduce the bug

```
# make sure that dataset decodes audio with correct sampling rate
dataset_sampling_rate = next(iter(self.raw_datasets.values())).features["audio"].sampling_rate
if dataset_sampling_rate != self.feature_extractor.sampling_rate:
    self.raw_datasets = self.raw_datasets.cast_column(
        "audio", datasets.features.Audio(sampling_rate=self.feature_extractor.sampling_rate)
    )

vectorized_datasets = self.raw_datasets.map(
    self.prepare_dataset,
    remove_columns=next(iter(self.raw_datasets.values())).column_names,
    num_proc=self.num_workers,
    desc="preprocess datasets",
)
# filter data that is longer than max_input_length
self.vectorized_datasets = vectorized_datasets.filter(
    self.is_audio_in_length_range,
    num_proc=self.num_workers,
    input_columns=["input_length"],
        )

def prepare_dataset(self, batch):
    # load audio
    sample = batch["audio"]
    inputs = self.feature_extractor(sample["array"], sampling_rate=sample["sampling_rate"])
    batch["input_values"] = inputs.input_values[0]
    batch["input_length"] = len(batch["input_values"])

    batch["labels"] = self.tokenizer(batch["target_text"]).input_ids
    return batch

```

### Expected behavior

`map` to use cached copy and if possible an alternative technique to reduce memory usage after using `map`

### Environment info

- `datasets` version: 2.12.0
- Platform: Linux-3.10.0-1160.71.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.16
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.2


loading private custom dataset script - authentication error

This issue seems to have been resolved, so I'm closing it.

### Describe the bug

Train model with my custom dataset stored in HuggingFace and loaded with the loading script requires authentication but I am not sure how ?

I am logged in in the terminal, in the browser. I receive this error:

/python3.8/site-packages/datasets/utils/file_utils.py", line 566, in get_from_cache
    raise ConnectionError(f"Couldn't reach {url} ({repr(head_error)})")
ConnectionError: Couldn't reach https://huggingface.co/datasets/fkov/s/blob/main/data/s/train/labels `(ConnectionError('Unauthorized for URL `https://huggingface.co/datasets/fkov/s/blob/main/data/s/train/labels. Please use the parameter `**`use_auth_token=True`**` after logging in with `**`huggingface-cli login`**`'))

when I added: `use_auth_token=True` and logged in via terminal then I received error:

or the same error in different format:
raise ConnectionError(f"`Couldn't reach {url} (error {response.status_code}`)")
ConnectionError: Couldn't reach https://huggingface.co/datasets/fkov/s/blob/main/data/s/train/labels (`error 401`)

### Steps to reproduce the bug

1.  cloned transformers library locally:
https://huggingface.co/docs/transformers/v4.15.0/examples :

> git clone https://github.com/huggingface/transformers
> cd transformers
> pip install .
> cd /transformers/examples/pytorch/audio-classification
> pip install -r requirements.txt

2. created **loading script**
> https://huggingface.co/docs/datasets/dataset_script  added next to dataset:

3. uploaded **private custom dataset** with loading script to HuggingFace
> https://huggingface.co/docs/datasets/dataset_script

4. added dataset loading script to **local directory** in the above cloned transformers library:
> cd /transformers/examples/pytorch/audio-classification

5. logged in to HuggingFace on local terminal with :
> **huggingface-cli login**

6.  run the model with the custom dataset stored on HuggingFace with code:  https://github.com/huggingface/transformers/blob/main/examples/pytorch/audio-classification/README.md

     cd /transformers/examples/pytorch/audio-classification
>  python run_audio_classification.py \
>     --model_name_or_path facebook/wav2vec2-base \
>     --output_dir l/users/flck/outputs/wav2vec2-base-s \
>     --overwrite_output_dir \
>     --dataset_name s \
>     --dataset_config_name s \
>     --remove_unused_columns False \
>     --do_train \
>     --do_eval \
>     --fp16 \
>     --learning_rate 3e-5 \
>     --max_length_seconds 1 \
>     --attention_mask False \
>     --warmup_ratio 0.1 \
>     --num_train_epochs 5 \
>     --per_device_train_batch_size 32 \
>     --gradient_accumulation_steps 4 \
>     --per_device_eval_batch_size 32 \
>     --dataloader_num_workers 4 \
>     --logging_strategy steps \
>     --logging_steps 10 \
>     --evaluation_strategy epoch \
>     --save_strategy epoch \
>     --load_best_model_at_end True \
>     --metric_for_best_model accuracy \
>     --save_total_limit 3 \
>     --seed 0 \
>     --push_to_hub \
>     **--use_auth_token=True**

### Expected behavior

Be able to train a model the https://github.com/huggingface/transformers/blob/main/examples/pytorch/audio-classification/ run_audio_classification.py with private custom dataset stored on HuggingFace.

### Environment info

- datasets version:                2.12.0
- `transformers` version: 4.30.0.dev0
- Platform: Linux-5.4.204-ql-generic-12.0-19-x86_64-with-glibc2.17
- Python version: 3.8.12
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu117 (True)

Versions of relevant libraries:
[pip3] numpy==1.24.3
[pip3] torch==2.0.1
[pip3] torchaudio==2.0.2
[conda] numpy                     1.24.3                   pypi_0    pypi
[conda] torch                     2.0.1                    pypi_0    pypi
[conda] torchaudio                2.0.2                    pypi_0    pypi


Importing PyTorch reduces multiprocessing performance for map

Hi! The times match when I run this code locally or on Colab.

Also, we use `multiprocess`, not `multiprocessing`, for parallelization, and torch's `__init__.py` (executed on `import torch` ) slightly modifies the latter.

Hey Mariosasko,

Thanks for looking into it. We further did some investigations after your comment and figured out it's only affecting some hardware/software configurations with the `pytorch` installation of `conda-forge`. Based on this we found the following issue in PyTorch: https://github.com/pytorch/pytorch/issues/102269 with a quick fix for now.

Since it seems to be a deeper issue with forking processes, the difference between`multiprocess` and `multiprocessing` didn't make a difference.

Closing this, since the issue comes from `pytorch` not `dataset`.

### Describe the bug

I noticed that the performance of my dataset preprocessing with `map(...,num_proc=32)` decreases when PyTorch is imported.

### Steps to reproduce the bug

I created two example scripts to reproduce this behavior:

```
import datasets
datasets.disable_caching()

from datasets import Dataset
import time

PROC=32

if __name__ == "__main__":
    dataset = [True] * 10000000
    dataset = Dataset.from_dict({'train': dataset})

    start = time.time()
    dataset.map(lambda x: x, num_proc=PROC)
    end = time.time()
    print(end - start)
```
Takes around 4 seconds on my machine.

While the same code, but with an `import torch`:
```
import datasets
datasets.disable_caching()

from datasets import Dataset
import time
import torch

PROC=32

if __name__ == "__main__":
    dataset = [True] * 10000000
    dataset = Dataset.from_dict({'train': dataset})

    start = time.time()
    dataset.map(lambda x: x, num_proc=PROC)
    end = time.time()
    print(end - start)
```
takes around 22 seconds.

### Expected behavior

I would expect that the import of torch to not have such a significant effect on the performance of map using multiprocessing.

### Environment info

- `datasets` version: 2.12.0
- Platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.35
- Python version: 3.11.3
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.2
- torch: 2.0.1


Fix link to quickstart docs in README.md

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006693 / 0.011353 (-0.004660) | 0.004331 / 0.011008 (-0.006677) | 0.098022 / 0.038508 (0.059514) | 0.032764 / 0.023109 (0.009654) | 0.295812 / 0.275898 (0.019914) | 0.325029 / 0.323480 (0.001550) | 0.005779 / 0.007986 (-0.002206) | 0.005381 / 0.004328 (0.001052) | 0.075785 / 0.004250 (0.071535) | 0.048759 / 0.037052 (0.011707) | 0.308986 / 0.258489 (0.050497) | 0.348000 / 0.293841 (0.054159) | 0.027686 / 0.128546 (-0.100860) | 0.008839 / 0.075646 (-0.066807) | 0.328389 / 0.419271 (-0.090883) | 0.062173 / 0.043533 (0.018640) | 0.312257 / 0.255139 (0.057119) | 0.325024 / 0.283200 (0.041824) | 0.103886 / 0.141683 (-0.037797) | 1.440215 / 1.452155 (-0.011940) | 1.528665 / 1.492716 (0.035948) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.210082 / 0.018006 (0.192076) | 0.442480 / 0.000490 (0.441990) | 0.006559 / 0.000200 (0.006359) | 0.000092 / 0.000054 (0.000038) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026774 / 0.037411 (-0.010637) | 0.108362 / 0.014526 (0.093837) | 0.117631 / 0.176557 (-0.058926) | 0.176657 / 0.737135 (-0.560478) | 0.124154 / 0.296338 (-0.172184) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.428136 / 0.215209 (0.212927) | 4.270287 / 2.077655 (2.192632) | 2.014728 / 1.504120 (0.510608) | 1.806772 / 1.541195 (0.265577) | 1.946284 / 1.468490 (0.477794) | 0.525542 / 4.584777 (-4.059235) | 3.667025 / 3.745712 (-0.078687) | 1.878751 / 5.269862 (-3.391111) | 1.048321 / 4.565676 (-3.517356) | 0.065550 / 0.424275 (-0.358725) | 0.011881 / 0.007607 (0.004274) | 0.529873 / 0.226044 (0.303829) | 5.289641 / 2.268929 (3.020712) | 2.489403 / 55.444624 (-52.955221) | 2.141037 / 6.876477 (-4.735440) | 2.230735 / 2.142072 (0.088662) | 0.639781 / 4.805227 (-4.165447) | 0.141410 / 6.500664 (-6.359254) | 0.064374 / 0.075469 (-0.011095) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.159462 / 1.841788 (-0.682325) | 14.524730 / 8.074308 (6.450422) | 13.578070 / 10.191392 (3.386678) | 0.152138 / 0.680424 (-0.528286) | 0.017255 / 0.534201 (-0.516946) | 0.387607 / 0.579283 (-0.191676) | 0.413652 / 0.434364 (-0.020712) | 0.453644 / 0.540337 (-0.086693) | 0.550051 / 1.386936 (-0.836885) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006668 / 0.011353 (-0.004685) | 0.004677 / 0.011008 (-0.006331) | 0.075950 / 0.038508 (0.037442) | 0.032439 / 0.023109 (0.009329) | 0.381839 / 0.275898 (0.105941) | 0.419411 / 0.323480 (0.095931) | 0.005813 / 0.007986 (-0.002172) | 0.004090 / 0.004328 (-0.000238) | 0.075052 / 0.004250 (0.070802) | 0.048453 / 0.037052 (0.011401) | 0.388076 / 0.258489 (0.129587) | 0.431793 / 0.293841 (0.137952) | 0.028408 / 0.128546 (-0.100138) | 0.009028 / 0.075646 (-0.066618) | 0.082569 / 0.419271 (-0.336702) | 0.046772 / 0.043533 (0.003239) | 0.380182 / 0.255139 (0.125043) | 0.401828 / 0.283200 (0.118629) | 0.105388 / 0.141683 (-0.036294) | 1.453356 / 1.452155 (0.001201) | 1.561483 / 1.492716 (0.068767) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.008922 / 0.018006 (-0.009084) | 0.444112 / 0.000490 (0.443623) | 0.002756 / 0.000200 (0.002556) | 0.000104 / 0.000054 (0.000050) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030408 / 0.037411 (-0.007003) | 0.112924 / 0.014526 (0.098399) | 0.124625 / 0.176557 (-0.051932) | 0.176915 / 0.737135 (-0.560220) | 0.129141 / 0.296338 (-0.167198) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.448197 / 0.215209 (0.232987) | 4.476548 / 2.077655 (2.398893) | 2.243977 / 1.504120 (0.739857) | 2.054060 / 1.541195 (0.512865) | 2.130680 / 1.468490 (0.662190) | 0.526815 / 4.584777 (-4.057962) | 3.759312 / 3.745712 (0.013600) | 3.333618 / 5.269862 (-1.936244) | 1.579611 / 4.565676 (-2.986065) | 0.065714 / 0.424275 (-0.358561) | 0.011939 / 0.007607 (0.004332) | 0.550313 / 0.226044 (0.324269) | 5.476946 / 2.268929 (3.208018) | 2.726521 / 55.444624 (-52.718104) | 2.364977 / 6.876477 (-4.511499) | 2.450624 / 2.142072 (0.308551) | 0.647174 / 4.805227 (-4.158053) | 0.141265 / 6.500664 (-6.359399) | 0.065493 / 0.075469 (-0.009976) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.249702 / 1.841788 (-0.592085) | 15.205647 / 8.074308 (7.131338) | 14.678310 / 10.191392 (4.486918) | 0.141539 / 0.680424 (-0.538884) | 0.017323 / 0.534201 (-0.516878) | 0.387602 / 0.579283 (-0.191681) | 0.415106 / 0.434364 (-0.019258) | 0.458146 / 0.540337 (-0.082192) | 0.553318 / 1.386936 (-0.833618) |

![](https://cml.dev/watermark.png#55127d7bf399fd2f3a8713db9822e8cb47cdbbed "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008567 / 0.011353 (-0.002786) | 0.005245 / 0.011008 (-0.005763) | 0.115074 / 0.038508 (0.076566) | 0.032567 / 0.023109 (0.009458) | 0.352297 / 0.275898 (0.076399) | 0.393403 / 0.323480 (0.069923) | 0.006402 / 0.007986 (-0.001583) | 0.004353 / 0.004328 (0.000025) | 0.087903 / 0.004250 (0.083653) | 0.048424 / 0.037052 (0.011372) | 0.370078 / 0.258489 (0.111588) | 0.410192 / 0.293841 (0.116351) | 0.042396 / 0.128546 (-0.086150) | 0.014426 / 0.075646 (-0.061220) | 0.411358 / 0.419271 (-0.007914) | 0.059546 / 0.043533 (0.016013) | 0.364721 / 0.255139 (0.109582) | 0.385100 / 0.283200 (0.101901) | 0.100572 / 0.141683 (-0.041111) | 1.741457 / 1.452155 (0.289302) | 1.933134 / 1.492716 (0.440418) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.217177 / 0.018006 (0.199171) | 0.510399 / 0.000490 (0.509909) | 0.005542 / 0.000200 (0.005342) | 0.000120 / 0.000054 (0.000065) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026852 / 0.037411 (-0.010559) | 0.125580 / 0.014526 (0.111054) | 0.132164 / 0.176557 (-0.044392) | 0.189073 / 0.737135 (-0.548063) | 0.135980 / 0.296338 (-0.160358) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.601924 / 0.215209 (0.386715) | 5.891397 / 2.077655 (3.813743) | 2.389494 / 1.504120 (0.885375) | 2.044013 / 1.541195 (0.502818) | 2.019367 / 1.468490 (0.550877) | 0.883807 / 4.584777 (-3.700970) | 5.141349 / 3.745712 (1.395636) | 2.607415 / 5.269862 (-2.662446) | 1.567268 / 4.565676 (-2.998409) | 0.102738 / 0.424275 (-0.321537) | 0.013480 / 0.007607 (0.005873) | 0.744979 / 0.226044 (0.518934) | 7.404182 / 2.268929 (5.135254) | 2.983406 / 55.444624 (-52.461219) | 2.331847 / 6.876477 (-4.544630) | 2.465119 / 2.142072 (0.323047) | 1.106725 / 4.805227 (-3.698502) | 0.205779 / 6.500664 (-6.294885) | 0.081019 / 0.075469 (0.005550) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.527840 / 1.841788 (-0.313947) | 16.989487 / 8.074308 (8.915179) | 18.016123 / 10.191392 (7.824731) | 0.216157 / 0.680424 (-0.464266) | 0.025393 / 0.534201 (-0.508808) | 0.496743 / 0.579283 (-0.082540) | 0.575365 / 0.434364 (0.141002) | 0.559978 / 0.540337 (0.019641) | 0.677474 / 1.386936 (-0.709462) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008913 / 0.011353 (-0.002440) | 0.005540 / 0.011008 (-0.005469) | 0.100001 / 0.038508 (0.061493) | 0.034432 / 0.023109 (0.011323) | 0.419824 / 0.275898 (0.143926) | 0.443566 / 0.323480 (0.120086) | 0.006372 / 0.007986 (-0.001614) | 0.004405 / 0.004328 (0.000077) | 0.094927 / 0.004250 (0.090677) | 0.050300 / 0.037052 (0.013248) | 0.424806 / 0.258489 (0.166317) | 0.480793 / 0.293841 (0.186952) | 0.050869 / 0.128546 (-0.077677) | 0.015899 / 0.075646 (-0.059747) | 0.111413 / 0.419271 (-0.307859) | 0.058093 / 0.043533 (0.014560) | 0.430575 / 0.255139 (0.175436) | 0.483786 / 0.283200 (0.200586) | 0.106878 / 0.141683 (-0.034805) | 1.763576 / 1.452155 (0.311422) | 1.837750 / 1.492716 (0.345033) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.011565 / 0.018006 (-0.006441) | 0.484411 / 0.000490 (0.483922) | 0.004869 / 0.000200 (0.004669) | 0.000111 / 0.000054 (0.000057) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030706 / 0.037411 (-0.006706) | 0.126901 / 0.014526 (0.112375) | 0.130367 / 0.176557 (-0.046190) | 0.206568 / 0.737135 (-0.530567) | 0.146505 / 0.296338 (-0.149834) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.627266 / 0.215209 (0.412057) | 6.314049 / 2.077655 (4.236394) | 2.582920 / 1.504120 (1.078800) | 2.249401 / 1.541195 (0.708206) | 2.244960 / 1.468490 (0.776470) | 0.907770 / 4.584777 (-3.677007) | 5.349622 / 3.745712 (1.603910) | 4.591244 / 5.269862 (-0.678618) | 2.301612 / 4.565676 (-2.264064) | 0.108813 / 0.424275 (-0.315462) | 0.013187 / 0.007607 (0.005580) | 0.806071 / 0.226044 (0.580027) | 7.843903 / 2.268929 (5.574974) | 3.405968 / 55.444624 (-52.038656) | 2.564301 / 6.876477 (-4.312176) | 2.652208 / 2.142072 (0.510135) | 1.168142 / 4.805227 (-3.637086) | 0.218551 / 6.500664 (-6.282113) | 0.078120 / 0.075469 (0.002651) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.562517 / 1.841788 (-0.279271) | 17.519325 / 8.074308 (9.445017) | 20.727083 / 10.191392 (10.535691) | 0.207135 / 0.680424 (-0.473288) | 0.028208 / 0.534201 (-0.505993) | 0.496157 / 0.579283 (-0.083126) | 0.569239 / 0.434364 (0.134875) | 0.566137 / 0.540337 (0.025799) | 0.704208 / 1.386936 (-0.682728) |

![](https://cml.dev/watermark.png#8eb3f34d876da98e722d866be90d7f26135ea9e3 "CML watermark")


`IndexError` when indexing `Sequence` of `Array2D` with `None` values

Easy fix would be to add:

```python
null_indices -= np.arange(len(null_indices))
```

before L279, but I'm not sure it's the most intuitive way to fix it.

Same issue here:

https://github.com/huggingface/datasets/blob/7fcbe5b1575c8d162b65b9397b3dfda995a4e048/src/datasets/features/features.py#L1398

Fixed in #5948

### Describe the bug

Having `None` values in a `Sequence` of `ArrayND` fails.

### Steps to reproduce the bug

```python
from datasets import Array2D, Dataset, Features, Sequence

data = [
    [
        [[0]],
        None,
        None,
    ]
]
feature = Sequence(Array2D((1, 1), dtype="int64"))
dataset = Dataset.from_dict({"a": data}, features=Features({"a": feature}))

dataset[0]  # error raised only when indexing
```

```
Traceback (most recent call last):
  File "/Users/quentingallouedec/gia/c.py", line 13, in <module>
    dataset[0]  # error raised only when indexing
  File "/Users/quentingallouedec/gia/env/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2658, in __getitem__
    return self._getitem(key)
  File "/Users/quentingallouedec/gia/env/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2643, in _getitem
    formatted_output = format_table(
  File "/Users/quentingallouedec/gia/env/lib/python3.10/site-packages/datasets/formatting/formatting.py", line 634, in format_table
    return formatter(pa_table, query_type=query_type)
  File "/Users/quentingallouedec/gia/env/lib/python3.10/site-packages/datasets/formatting/formatting.py", line 406, in __call__
    return self.format_row(pa_table)
  File "/Users/quentingallouedec/gia/env/lib/python3.10/site-packages/datasets/formatting/formatting.py", line 441, in format_row
    row = self.python_arrow_extractor().extract_row(pa_table)
  File "/Users/quentingallouedec/gia/env/lib/python3.10/site-packages/datasets/formatting/formatting.py", line 144, in extract_row
    return _unnest(pa_table.to_pydict())
  File "pyarrow/table.pxi", line 4146, in pyarrow.lib.Table.to_pydict
  File "pyarrow/table.pxi", line 1312, in pyarrow.lib.ChunkedArray.to_pylist
  File "pyarrow/array.pxi", line 1521, in pyarrow.lib.Array.to_pylist
  File "pyarrow/scalar.pxi", line 675, in pyarrow.lib.ListScalar.as_py
  File "/Users/quentingallouedec/gia/env/lib/python3.10/site-packages/datasets/features/features.py", line 760, in to_pylist
    return self.to_numpy(zero_copy_only=zero_copy_only).tolist()
  File "/Users/quentingallouedec/gia/env/lib/python3.10/site-packages/datasets/features/features.py", line 725, in to_numpy
    numpy_arr = np.insert(numpy_arr.astype(np.float64), null_indices, np.nan, axis=0)
  File "<__array_function__ internals>", line 200, in insert
  File "/Users/quentingallouedec/gia/env/lib/python3.10/site-packages/numpy/lib/function_base.py", line 5426, in insert
    old_mask[indices] = False
IndexError: index 3 is out of bounds for axis 0 with size 3
```

AFAIK, the problem only occurs when you use a `Sequence` of `ArrayND`.

I strongly suspect that the problem comes from this line, or `np.insert` is misused:

https://github.com/huggingface/datasets/blob/02ee418831aba68d0be93227bce8b3f42ef8980f/src/datasets/features/features.py#L729

To put t simply, you want something that do that:

```python
import numpy as np
numpy_arr = np.zeros((1, 1, 1))
null_indices = np.array([1, 2])
np.insert(numpy_arr, null_indices, np.nan, axis=0)
# raise an error, instead of outputting
# array([[[ 0.]],
#        [[nan]],
#        [[nan]]])
```

### Expected behavior

The previous code should not raise an error.

### Environment info

- Python 3.10.11
- datasets 2.10.0
- pyarrow 12.0.0


Uncaught exception when generating the splits from a dataset that miss data

Thanks for reporting, @severo.

This is a known issue with `fsspec`:
- #5862
- https://github.com/fsspec/filesystem_spec/issues/1265

### Describe the bug

Dataset https://huggingface.co/datasets/blog_authorship_corpus has an issue with its hosting platform, since https://drive.google.com/u/0/uc?id=1cGy4RNDV87ZHEXbiozABr9gsSrZpPaPz&export=download returns 404 error.

But when trying to generate the split names, we get an exception which is now correctly caught.

Seen originally in https://github.com/huggingface/datasets-server/blob/adbdcd6710ffed4e2eb2e4cd905b5e0dff530a15/services/worker/src/worker/job_runners/config/parquet_and_info.py#L435

### Steps to reproduce the bug

```python
>>> from datasets import StreamingDownloadManager, load_dataset_builder
>>> builder = load_dataset_builder(path="blog_authorship_corpus")
Downloading builder script: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5.60k/5.60k [00:00<00:00, 23.1MB/s]
Downloading metadata: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.81k/2.81k [00:00<00:00, 14.7MB/s]
Downloading readme: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7.30k/7.30k [00:00<00:00, 30.8MB/s]
>>> dl_manager = StreamingDownloadManager(base_path=builder.base_path)
>>> builder._split_generators(dl_manager)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/blog_authorship_corpus/6f5d78241afd8313111956f877a57db7a0e9fc6718255dc85df0928197feb683/blog_authorship_corpus.py", line 79, in _split_generators
    data = dl_manager.download_and_extract(_DATA_URL)
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/download/streaming_download_manager.py", line 1087, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/download/streaming_download_manager.py", line 1039, in extract
    urlpaths = map_nested(self._extract, url_or_urls, map_tuple=True)
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 435, in map_nested
    return function(data_struct)
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/download/streaming_download_manager.py", line 1044, in _extract
    protocol = _get_extraction_protocol(urlpath, use_auth_token=self.download_config.use_auth_token)
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/download/streaming_download_manager.py", line 433, in _get_extraction_protocol
    with fsspec.open(urlpath, **kwargs) as f:
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/fsspec/core.py", line 439, in open
    return open_files(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/fsspec/core.py", line 194, in __getitem__
    out = super().__getitem__(item)
IndexError: list index out of range
```

### Expected behavior

We should have an Exception raised by the datasets library.

### Environment info

- `datasets` version: 2.12.0
- Platform: Linux-5.19.0-1026-aws-x86_64-with-glibc2.35
- Python version: 3.9.15
- Huggingface_hub version: 0.15.1
- PyArrow version: 11.0.0
- Pandas version: 2.0.2


Breaking API change in datasets.list_datasets caused by change in HfApi.list_datasets

### Describe the bug

Hi all,

after an update of the `datasets` library, we observer crashes in our code. We relied on `datasets.list_datasets` returning a `list`. Now, after the API of the HfApi.list_datasets was changed and it returns a `list` instead of an `Iterable`, the `datasets.list_datasets` now sometimes returns a `list` and somesimes an `Iterable`.

It would be helpful to indicate that by the return type of the `datasets.list_datasets` function.

Thanks,
Martin

### Steps to reproduce the bug

Here, the code crashed after we updated the `datasets` library:

```python
# list_datasets no longer returns a list, which leads to an error when one tries to slice it
for datasets.list_datasets(with_details=True)[:limit]:
    ...
```

### Expected behavior

It would be helpful to indicate that by the return type of the `datasets.list_datasets` function.

### Environment info

Ubuntu 22.04
datasets 2.12.0