set dev version

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5979). All of your documentation changes will be reflected on that endpoint.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008087 / 0.011353 (-0.003266) | 0.004691 / 0.011008 (-0.006317) | 0.121545 / 0.038508 (0.083037) | 0.057436 / 0.023109 (0.034326) | 0.368864 / 0.275898 (0.092966) | 0.457199 / 0.323480 (0.133719) | 0.006745 / 0.007986 (-0.001241) | 0.003689 / 0.004328 (-0.000640) | 0.090480 / 0.004250 (0.086229) | 0.071368 / 0.037052 (0.034316) | 0.372788 / 0.258489 (0.114299) | 0.429894 / 0.293841 (0.136053) | 0.037544 / 0.128546 (-0.091002) | 0.010142 / 0.075646 (-0.065505) | 0.420467 / 0.419271 (0.001196) | 0.064359 / 0.043533 (0.020826) | 0.370345 / 0.255139 (0.115206) | 0.405220 / 0.283200 (0.122020) | 0.028410 / 0.141683 (-0.113273) | 1.824845 / 1.452155 (0.372690) | 1.888109 / 1.492716 (0.395392) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.234585 / 0.018006 (0.216578) | 0.499965 / 0.000490 (0.499476) | 0.000461 / 0.000200 (0.000261) | 0.000064 / 0.000054 (0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032294 / 0.037411 (-0.005117) | 0.131769 / 0.014526 (0.117243) | 0.146472 / 0.176557 (-0.030085) | 0.210035 / 0.737135 (-0.527100) | 0.145600 / 0.296338 (-0.150739) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.507455 / 0.215209 (0.292246) | 5.080090 / 2.077655 (3.002435) | 2.506104 / 1.504120 (1.001984) | 2.297655 / 1.541195 (0.756460) | 2.324920 / 1.468490 (0.856430) | 0.645003 / 4.584777 (-3.939774) | 4.677856 / 3.745712 (0.932144) | 2.254179 / 5.269862 (-3.015683) | 1.280663 / 4.565676 (-3.285013) | 0.078809 / 0.424275 (-0.345466) | 0.014059 / 0.007607 (0.006452) | 0.628053 / 0.226044 (0.402009) | 6.327289 / 2.268929 (4.058360) | 2.957918 / 55.444624 (-52.486706) | 2.571568 / 6.876477 (-4.304909) | 2.708766 / 2.142072 (0.566694) | 0.772868 / 4.805227 (-4.032360) | 0.164835 / 6.500664 (-6.335829) | 0.075334 / 0.075469 (-0.000135) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.471930 / 1.841788 (-0.369858) | 17.917340 / 8.074308 (9.843032) | 15.719327 / 10.191392 (5.527935) | 0.191999 / 0.680424 (-0.488424) | 0.022464 / 0.534201 (-0.511737) | 0.511038 / 0.579283 (-0.068245) | 0.512050 / 0.434364 (0.077686) | 0.608711 / 0.540337 (0.068373) | 0.749660 / 1.386936 (-0.637276) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008028 / 0.011353 (-0.003325) | 0.004908 / 0.011008 (-0.006100) | 0.092294 / 0.038508 (0.053786) | 0.053051 / 0.023109 (0.029942) | 0.453862 / 0.275898 (0.177964) | 0.512548 / 0.323480 (0.189068) | 0.004817 / 0.007986 (-0.003168) | 0.005330 / 0.004328 (0.001002) | 0.095600 / 0.004250 (0.091350) | 0.068763 / 0.037052 (0.031710) | 0.453654 / 0.258489 (0.195165) | 0.504995 / 0.293841 (0.211154) | 0.038123 / 0.128546 (-0.090423) | 0.010650 / 0.075646 (-0.064996) | 0.102854 / 0.419271 (-0.316417) | 0.062973 / 0.043533 (0.019440) | 0.430420 / 0.255139 (0.175281) | 0.465448 / 0.283200 (0.182248) | 0.029736 / 0.141683 (-0.111947) | 1.844225 / 1.452155 (0.392070) | 1.934685 / 1.492716 (0.441968) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.227797 / 0.018006 (0.209791) | 0.467868 / 0.000490 (0.467378) | 0.004531 / 0.000200 (0.004331) | 0.000105 / 0.000054 (0.000051) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.035632 / 0.037411 (-0.001780) | 0.145943 / 0.014526 (0.131417) | 0.151944 / 0.176557 (-0.024613) | 0.220519 / 0.737135 (-0.516616) | 0.159732 / 0.296338 (-0.136606) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.520641 / 0.215209 (0.305432) | 5.184740 / 2.077655 (3.107086) | 2.538751 / 1.504120 (1.034631) | 2.316571 / 1.541195 (0.775377) | 2.387898 / 1.468490 (0.919408) | 0.614515 / 4.584777 (-3.970262) | 4.573142 / 3.745712 (0.827430) | 4.657052 / 5.269862 (-0.612809) | 2.159664 / 4.565676 (-2.406013) | 0.079713 / 0.424275 (-0.344562) | 0.014462 / 0.007607 (0.006855) | 0.656611 / 0.226044 (0.430566) | 6.481630 / 2.268929 (4.212702) | 3.135047 / 55.444624 (-52.309577) | 2.757502 / 6.876477 (-4.118975) | 2.851488 / 2.142072 (0.709415) | 0.790795 / 4.805227 (-4.014432) | 0.172358 / 6.500664 (-6.328306) | 0.080255 / 0.075469 (0.004786) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.571391 / 1.841788 (-0.270396) | 19.025224 / 8.074308 (10.950916) | 17.079230 / 10.191392 (6.887838) | 0.172823 / 0.680424 (-0.507601) | 0.021845 / 0.534201 (-0.512356) | 0.522286 / 0.579283 (-0.056998) | 0.510406 / 0.434364 (0.076042) | 0.604830 / 0.540337 (0.064493) | 0.735466 / 1.386936 (-0.651471) |

![](https://cml.dev/watermark.png#4084609bdc40d173d1daa74ad2fe98f3ead72f8e "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.010025 / 0.011353 (-0.001328) | 0.005699 / 0.011008 (-0.005310) | 0.134194 / 0.038508 (0.095686) | 0.056154 / 0.023109 (0.033045) | 0.470091 / 0.275898 (0.194193) | 0.539225 / 0.323480 (0.215745) | 0.006659 / 0.007986 (-0.001326) | 0.004468 / 0.004328 (0.000140) | 0.110040 / 0.004250 (0.105790) | 0.074172 / 0.037052 (0.037119) | 0.497450 / 0.258489 (0.238961) | 0.535048 / 0.293841 (0.241207) | 0.051195 / 0.128546 (-0.077352) | 0.014926 / 0.075646 (-0.060721) | 0.461334 / 0.419271 (0.042062) | 0.073773 / 0.043533 (0.030240) | 0.450741 / 0.255139 (0.195602) | 0.474853 / 0.283200 (0.191653) | 0.036372 / 0.141683 (-0.105311) | 1.982873 / 1.452155 (0.530719) | 1.989912 / 1.492716 (0.497196) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.287817 / 0.018006 (0.269811) | 0.613415 / 0.000490 (0.612926) | 0.007082 / 0.000200 (0.006882) | 0.000100 / 0.000054 (0.000045) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031119 / 0.037411 (-0.006292) | 0.129886 / 0.014526 (0.115361) | 0.143492 / 0.176557 (-0.033065) | 0.208536 / 0.737135 (-0.528600) | 0.147081 / 0.296338 (-0.149257) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.668312 / 0.215209 (0.453103) | 6.568609 / 2.077655 (4.490955) | 2.708788 / 1.504120 (1.204668) | 2.366737 / 1.541195 (0.825542) | 2.392598 / 1.468490 (0.924108) | 0.967582 / 4.584777 (-3.617195) | 5.582743 / 3.745712 (1.837031) | 3.021607 / 5.269862 (-2.248255) | 1.866402 / 4.565676 (-2.699275) | 0.115998 / 0.424275 (-0.308277) | 0.015571 / 0.007607 (0.007964) | 0.820069 / 0.226044 (0.594025) | 8.229725 / 2.268929 (5.960797) | 3.437068 / 55.444624 (-52.007557) | 2.902312 / 6.876477 (-3.974164) | 3.025874 / 2.142072 (0.883802) | 1.230359 / 4.805227 (-3.574868) | 0.237341 / 6.500664 (-6.263323) | 0.089923 / 0.075469 (0.014453) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.670970 / 1.841788 (-0.170818) | 19.667167 / 8.074308 (11.592859) | 21.624423 / 10.191392 (11.433031) | 0.231683 / 0.680424 (-0.448741) | 0.029145 / 0.534201 (-0.505056) | 0.543441 / 0.579283 (-0.035842) | 0.617510 / 0.434364 (0.183146) | 0.612662 / 0.540337 (0.072324) | 0.790589 / 1.386936 (-0.596347) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.010324 / 0.011353 (-0.001029) | 0.005339 / 0.011008 (-0.005669) | 0.104762 / 0.038508 (0.066254) | 0.052631 / 0.023109 (0.029522) | 0.485864 / 0.275898 (0.209966) | 0.595768 / 0.323480 (0.272288) | 0.007417 / 0.007986 (-0.000569) | 0.005229 / 0.004328 (0.000900) | 0.100775 / 0.004250 (0.096524) | 0.067144 / 0.037052 (0.030092) | 0.522269 / 0.258489 (0.263780) | 0.592597 / 0.293841 (0.298756) | 0.051101 / 0.128546 (-0.077446) | 0.015277 / 0.075646 (-0.060369) | 0.115530 / 0.419271 (-0.303741) | 0.071922 / 0.043533 (0.028390) | 0.490208 / 0.255139 (0.235069) | 0.578936 / 0.283200 (0.295736) | 0.040382 / 0.141683 (-0.101301) | 1.986059 / 1.452155 (0.533904) | 2.040600 / 1.492716 (0.547883) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.300399 / 0.018006 (0.282393) | 0.624702 / 0.000490 (0.624212) | 0.004908 / 0.000200 (0.004708) | 0.000155 / 0.000054 (0.000100) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.038031 / 0.037411 (0.000619) | 0.140353 / 0.014526 (0.125828) | 0.152600 / 0.176557 (-0.023956) | 0.219165 / 0.737135 (-0.517970) | 0.154232 / 0.296338 (-0.142106) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.698855 / 0.215209 (0.483646) | 7.125543 / 2.077655 (5.047889) | 3.251222 / 1.504120 (1.747102) | 2.953404 / 1.541195 (1.412209) | 3.051108 / 1.468490 (1.582618) | 0.962068 / 4.584777 (-3.622709) | 5.789579 / 3.745712 (2.043867) | 5.193271 / 5.269862 (-0.076591) | 2.757886 / 4.565676 (-1.807790) | 0.111865 / 0.424275 (-0.312410) | 0.014684 / 0.007607 (0.007077) | 0.875967 / 0.226044 (0.649923) | 8.818359 / 2.268929 (6.549430) | 4.165216 / 55.444624 (-51.279408) | 3.372059 / 6.876477 (-3.504418) | 3.486886 / 2.142072 (1.344813) | 1.232276 / 4.805227 (-3.572951) | 0.238967 / 6.500664 (-6.261697) | 0.091584 / 0.075469 (0.016115) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.850755 / 1.841788 (0.008968) | 20.058756 / 8.074308 (11.984448) | 23.761271 / 10.191392 (13.569879) | 0.231826 / 0.680424 (-0.448598) | 0.030119 / 0.534201 (-0.504082) | 0.532614 / 0.579283 (-0.046669) | 0.628968 / 0.434364 (0.194604) | 0.628403 / 0.540337 (0.088066) | 0.745648 / 1.386936 (-0.641288) |

![](https://cml.dev/watermark.png#a8a797cc92e860c8d0df71e0aa826f4d2690713e "CML watermark")


Release: 2.13.1

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006173 / 0.011353 (-0.005180) | 0.003773 / 0.011008 (-0.007235) | 0.099499 / 0.038508 (0.060991) | 0.037918 / 0.023109 (0.014809) | 0.321329 / 0.275898 (0.045431) | 0.379739 / 0.323480 (0.056259) | 0.004664 / 0.007986 (-0.003322) | 0.002943 / 0.004328 (-0.001385) | 0.077759 / 0.004250 (0.073509) | 0.055271 / 0.037052 (0.018219) | 0.329428 / 0.258489 (0.070939) | 0.378731 / 0.293841 (0.084890) | 0.027737 / 0.128546 (-0.100810) | 0.008566 / 0.075646 (-0.067081) | 0.313220 / 0.419271 (-0.106052) | 0.047101 / 0.043533 (0.003568) | 0.316211 / 0.255139 (0.061072) | 0.341826 / 0.283200 (0.058626) | 0.020838 / 0.141683 (-0.120845) | 1.550064 / 1.452155 (0.097909) | 1.706518 / 1.492716 (0.213801) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.203093 / 0.018006 (0.185087) | 0.425345 / 0.000490 (0.424856) | 0.004800 / 0.000200 (0.004600) | 0.000077 / 0.000054 (0.000022) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024590 / 0.037411 (-0.012821) | 0.098115 / 0.014526 (0.083589) | 0.108274 / 0.176557 (-0.068282) | 0.170804 / 0.737135 (-0.566332) | 0.110560 / 0.296338 (-0.185778) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.425251 / 0.215209 (0.210042) | 4.239075 / 2.077655 (2.161421) | 1.955601 / 1.504120 (0.451481) | 1.774796 / 1.541195 (0.233602) | 1.826641 / 1.468490 (0.358150) | 0.558777 / 4.584777 (-4.026000) | 3.361697 / 3.745712 (-0.384015) | 1.764468 / 5.269862 (-3.505394) | 1.032280 / 4.565676 (-3.533396) | 0.067872 / 0.424275 (-0.356403) | 0.010998 / 0.007607 (0.003391) | 0.525682 / 0.226044 (0.299637) | 5.254356 / 2.268929 (2.985427) | 2.384332 / 55.444624 (-53.060292) | 2.045578 / 6.876477 (-4.830898) | 2.170914 / 2.142072 (0.028841) | 0.674782 / 4.805227 (-4.130445) | 0.135351 / 6.500664 (-6.365314) | 0.066591 / 0.075469 (-0.008878) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.209181 / 1.841788 (-0.632606) | 14.044518 / 8.074308 (5.970210) | 13.184705 / 10.191392 (2.993313) | 0.130836 / 0.680424 (-0.549588) | 0.016582 / 0.534201 (-0.517619) | 0.360005 / 0.579283 (-0.219279) | 0.379519 / 0.434364 (-0.054845) | 0.422174 / 0.540337 (-0.118164) | 0.515546 / 1.386936 (-0.871390) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006293 / 0.011353 (-0.005060) | 0.003784 / 0.011008 (-0.007224) | 0.079248 / 0.038508 (0.040739) | 0.038452 / 0.023109 (0.015343) | 0.444727 / 0.275898 (0.168829) | 0.500535 / 0.323480 (0.177055) | 0.003455 / 0.007986 (-0.004531) | 0.002873 / 0.004328 (-0.001455) | 0.077439 / 0.004250 (0.073189) | 0.047855 / 0.037052 (0.010803) | 0.448049 / 0.258489 (0.189560) | 0.509517 / 0.293841 (0.215676) | 0.028359 / 0.128546 (-0.100188) | 0.008503 / 0.075646 (-0.067143) | 0.084961 / 0.419271 (-0.334310) | 0.042880 / 0.043533 (-0.000653) | 0.436628 / 0.255139 (0.181489) | 0.456574 / 0.283200 (0.173375) | 0.019539 / 0.141683 (-0.122144) | 1.561273 / 1.452155 (0.109118) | 1.572018 / 1.492716 (0.079301) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.230250 / 0.018006 (0.212244) | 0.415189 / 0.000490 (0.414700) | 0.003213 / 0.000200 (0.003013) | 0.000080 / 0.000054 (0.000026) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025541 / 0.037411 (-0.011871) | 0.102326 / 0.014526 (0.087800) | 0.110258 / 0.176557 (-0.066298) | 0.162488 / 0.737135 (-0.574647) | 0.112782 / 0.296338 (-0.183556) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.457936 / 0.215209 (0.242727) | 4.581503 / 2.077655 (2.503848) | 2.237659 / 1.504120 (0.733540) | 2.029960 / 1.541195 (0.488765) | 2.082911 / 1.468490 (0.614421) | 0.556485 / 4.584777 (-4.028292) | 3.384418 / 3.745712 (-0.361295) | 1.748809 / 5.269862 (-3.521053) | 1.034759 / 4.565676 (-3.530917) | 0.067500 / 0.424275 (-0.356776) | 0.011425 / 0.007607 (0.003818) | 0.561340 / 0.226044 (0.335295) | 5.623629 / 2.268929 (3.354701) | 2.733587 / 55.444624 (-52.711038) | 2.401578 / 6.876477 (-4.474899) | 2.524569 / 2.142072 (0.382496) | 0.673170 / 4.805227 (-4.132057) | 0.136681 / 6.500664 (-6.363983) | 0.068060 / 0.075469 (-0.007409) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.318651 / 1.841788 (-0.523137) | 14.362123 / 8.074308 (6.287815) | 14.385964 / 10.191392 (4.194572) | 0.149914 / 0.680424 (-0.530510) | 0.016877 / 0.534201 (-0.517324) | 0.358406 / 0.579283 (-0.220877) | 0.394349 / 0.434364 (-0.040015) | 0.422471 / 0.540337 (-0.117866) | 0.513807 / 1.386936 (-0.873129) |

![](https://cml.dev/watermark.png#1b9ce11d1b94e6178df663ff5fcad029849d10fb "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006272 / 0.011353 (-0.005080) | 0.003903 / 0.011008 (-0.007105) | 0.100180 / 0.038508 (0.061672) | 0.037799 / 0.023109 (0.014690) | 0.385627 / 0.275898 (0.109729) | 0.446518 / 0.323480 (0.123038) | 0.004811 / 0.007986 (-0.003175) | 0.003032 / 0.004328 (-0.001296) | 0.077063 / 0.004250 (0.072812) | 0.055564 / 0.037052 (0.018512) | 0.397346 / 0.258489 (0.138857) | 0.443242 / 0.293841 (0.149401) | 0.027904 / 0.128546 (-0.100642) | 0.008386 / 0.075646 (-0.067260) | 0.315013 / 0.419271 (-0.104259) | 0.047943 / 0.043533 (0.004410) | 0.378443 / 0.255139 (0.123304) | 0.411472 / 0.283200 (0.128272) | 0.020465 / 0.141683 (-0.121218) | 1.526594 / 1.452155 (0.074439) | 1.547018 / 1.492716 (0.054301) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.219377 / 0.018006 (0.201370) | 0.430254 / 0.000490 (0.429764) | 0.003218 / 0.000200 (0.003018) | 0.000072 / 0.000054 (0.000017) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023667 / 0.037411 (-0.013744) | 0.099143 / 0.014526 (0.084617) | 0.106044 / 0.176557 (-0.070513) | 0.166186 / 0.737135 (-0.570949) | 0.108736 / 0.296338 (-0.187603) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.437971 / 0.215209 (0.222762) | 4.363675 / 2.077655 (2.286021) | 2.011993 / 1.504120 (0.507873) | 1.845189 / 1.541195 (0.303994) | 1.831848 / 1.468490 (0.363358) | 0.562402 / 4.584777 (-4.022375) | 3.365259 / 3.745712 (-0.380453) | 1.781491 / 5.269862 (-3.488371) | 1.023454 / 4.565676 (-3.542223) | 0.067857 / 0.424275 (-0.356418) | 0.011076 / 0.007607 (0.003469) | 0.532267 / 0.226044 (0.306223) | 5.340344 / 2.268929 (3.071415) | 2.388649 / 55.444624 (-53.055976) | 2.055373 / 6.876477 (-4.821104) | 2.205047 / 2.142072 (0.062975) | 0.672909 / 4.805227 (-4.132318) | 0.135244 / 6.500664 (-6.365420) | 0.066184 / 0.075469 (-0.009285) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.206838 / 1.841788 (-0.634950) | 13.967075 / 8.074308 (5.892767) | 13.143971 / 10.191392 (2.952579) | 0.143991 / 0.680424 (-0.536433) | 0.016673 / 0.534201 (-0.517527) | 0.376180 / 0.579283 (-0.203103) | 0.386550 / 0.434364 (-0.047814) | 0.440590 / 0.540337 (-0.099747) | 0.529974 / 1.386936 (-0.856962) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006299 / 0.011353 (-0.005054) | 0.003784 / 0.011008 (-0.007224) | 0.077875 / 0.038508 (0.039367) | 0.038689 / 0.023109 (0.015580) | 0.421684 / 0.275898 (0.145786) | 0.472649 / 0.323480 (0.149169) | 0.003570 / 0.007986 (-0.004415) | 0.004448 / 0.004328 (0.000120) | 0.077867 / 0.004250 (0.073616) | 0.049514 / 0.037052 (0.012462) | 0.375983 / 0.258489 (0.117494) | 0.470632 / 0.293841 (0.176791) | 0.028238 / 0.128546 (-0.100308) | 0.008462 / 0.075646 (-0.067185) | 0.082452 / 0.419271 (-0.336819) | 0.043617 / 0.043533 (0.000084) | 0.400874 / 0.255139 (0.145735) | 0.426191 / 0.283200 (0.142992) | 0.020602 / 0.141683 (-0.121081) | 1.567658 / 1.452155 (0.115504) | 1.572610 / 1.492716 (0.079893) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.246144 / 0.018006 (0.228138) | 0.419402 / 0.000490 (0.418913) | 0.001691 / 0.000200 (0.001491) | 0.000071 / 0.000054 (0.000017) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026105 / 0.037411 (-0.011306) | 0.104734 / 0.014526 (0.090208) | 0.110257 / 0.176557 (-0.066300) | 0.161429 / 0.737135 (-0.575706) | 0.114367 / 0.296338 (-0.181972) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.453352 / 0.215209 (0.238143) | 4.537924 / 2.077655 (2.460269) | 2.196193 / 1.504120 (0.692073) | 2.002087 / 1.541195 (0.460892) | 2.041722 / 1.468490 (0.573231) | 0.561643 / 4.584777 (-4.023134) | 3.449108 / 3.745712 (-0.296605) | 2.862800 / 5.269862 (-2.407062) | 1.387895 / 4.565676 (-3.177782) | 0.068076 / 0.424275 (-0.356199) | 0.011568 / 0.007607 (0.003961) | 0.559279 / 0.226044 (0.333235) | 5.598738 / 2.268929 (3.329809) | 2.676649 / 55.444624 (-52.767975) | 2.334588 / 6.876477 (-4.541889) | 2.376215 / 2.142072 (0.234142) | 0.673109 / 4.805227 (-4.132118) | 0.137587 / 6.500664 (-6.363077) | 0.069131 / 0.075469 (-0.006338) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.307332 / 1.841788 (-0.534456) | 14.536036 / 8.074308 (6.461728) | 14.173734 / 10.191392 (3.982342) | 0.145143 / 0.680424 (-0.535281) | 0.016662 / 0.534201 (-0.517539) | 0.366901 / 0.579283 (-0.212383) | 0.394498 / 0.434364 (-0.039866) | 0.430546 / 0.540337 (-0.109792) | 0.518950 / 1.386936 (-0.867986) |

![](https://cml.dev/watermark.png#682d21e94ab1e64c11b583de39dc4c93f0101c5a "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008122 / 0.011353 (-0.003231) | 0.005585 / 0.011008 (-0.005424) | 0.121219 / 0.038508 (0.082711) | 0.047616 / 0.023109 (0.024507) | 0.440576 / 0.275898 (0.164678) | 0.491053 / 0.323480 (0.167573) | 0.004774 / 0.007986 (-0.003211) | 0.006758 / 0.004328 (0.002430) | 0.103852 / 0.004250 (0.099602) | 0.071560 / 0.037052 (0.034508) | 0.463107 / 0.258489 (0.204618) | 0.516904 / 0.293841 (0.223063) | 0.048052 / 0.128546 (-0.080494) | 0.013679 / 0.075646 (-0.061968) | 0.428383 / 0.419271 (0.009112) | 0.069468 / 0.043533 (0.025936) | 0.432593 / 0.255139 (0.177454) | 0.471810 / 0.283200 (0.188611) | 0.037541 / 0.141683 (-0.104142) | 1.823490 / 1.452155 (0.371335) | 1.922558 / 1.492716 (0.429842) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.252315 / 0.018006 (0.234309) | 0.541757 / 0.000490 (0.541267) | 0.000373 / 0.000200 (0.000173) | 0.000083 / 0.000054 (0.000028) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030361 / 0.037411 (-0.007050) | 0.125928 / 0.014526 (0.111402) | 0.145102 / 0.176557 (-0.031455) | 0.209798 / 0.737135 (-0.527337) | 0.147349 / 0.296338 (-0.148990) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.627554 / 0.215209 (0.412345) | 5.917422 / 2.077655 (3.839767) | 2.491083 / 1.504120 (0.986963) | 2.147078 / 1.541195 (0.605883) | 2.167511 / 1.468490 (0.699021) | 0.903061 / 4.584777 (-3.681716) | 5.518537 / 3.745712 (1.772825) | 2.654348 / 5.269862 (-2.615514) | 1.645121 / 4.565676 (-2.920556) | 0.103782 / 0.424275 (-0.320493) | 0.013048 / 0.007607 (0.005441) | 0.756732 / 0.226044 (0.530687) | 7.622873 / 2.268929 (5.353945) | 3.122689 / 55.444624 (-52.321936) | 2.537735 / 6.876477 (-4.338742) | 2.640090 / 2.142072 (0.498018) | 1.128635 / 4.805227 (-3.676593) | 0.228089 / 6.500664 (-6.272575) | 0.086207 / 0.075469 (0.010738) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.561591 / 1.841788 (-0.280197) | 18.110299 / 8.074308 (10.035991) | 20.718017 / 10.191392 (10.526625) | 0.225741 / 0.680424 (-0.454682) | 0.031738 / 0.534201 (-0.502463) | 0.530789 / 0.579283 (-0.048495) | 0.607364 / 0.434364 (0.173000) | 0.581593 / 0.540337 (0.041256) | 0.726033 / 1.386936 (-0.660903) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009323 / 0.011353 (-0.002030) | 0.005360 / 0.011008 (-0.005649) | 0.103608 / 0.038508 (0.065100) | 0.050158 / 0.023109 (0.027049) | 0.499906 / 0.275898 (0.224008) | 0.561005 / 0.323480 (0.237525) | 0.005093 / 0.007986 (-0.002892) | 0.008285 / 0.004328 (0.003956) | 0.103446 / 0.004250 (0.099196) | 0.061478 / 0.037052 (0.024426) | 0.494016 / 0.258489 (0.235527) | 0.537550 / 0.293841 (0.243709) | 0.048829 / 0.128546 (-0.079717) | 0.017032 / 0.075646 (-0.058614) | 0.107748 / 0.419271 (-0.311524) | 0.065607 / 0.043533 (0.022074) | 0.488709 / 0.255139 (0.233570) | 0.512023 / 0.283200 (0.228823) | 0.032067 / 0.141683 (-0.109616) | 1.907585 / 1.452155 (0.455431) | 1.960994 / 1.492716 (0.468278) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.278378 / 0.018006 (0.260371) | 0.551474 / 0.000490 (0.550985) | 0.006886 / 0.000200 (0.006686) | 0.000106 / 0.000054 (0.000051) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030674 / 0.037411 (-0.006737) | 0.135179 / 0.014526 (0.120654) | 0.133703 / 0.176557 (-0.042853) | 0.198923 / 0.737135 (-0.538212) | 0.155108 / 0.296338 (-0.141231) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.690566 / 0.215209 (0.475357) | 6.789594 / 2.077655 (4.711940) | 2.940668 / 1.504120 (1.436549) | 2.562431 / 1.541195 (1.021236) | 2.554232 / 1.468490 (1.085742) | 0.888470 / 4.584777 (-3.696307) | 5.672318 / 3.745712 (1.926606) | 2.741626 / 5.269862 (-2.528236) | 1.818336 / 4.565676 (-2.747340) | 0.110434 / 0.424275 (-0.313841) | 0.014114 / 0.007607 (0.006507) | 0.830632 / 0.226044 (0.604588) | 8.270787 / 2.268929 (6.001859) | 3.723486 / 55.444624 (-51.721139) | 2.993671 / 6.876477 (-3.882806) | 2.918273 / 2.142072 (0.776201) | 1.105337 / 4.805227 (-3.699891) | 0.222976 / 6.500664 (-6.277688) | 0.085290 / 0.075469 (0.009820) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.816027 / 1.841788 (-0.025760) | 18.496850 / 8.074308 (10.422541) | 20.457032 / 10.191392 (10.265640) | 0.243533 / 0.680424 (-0.436891) | 0.027044 / 0.534201 (-0.507157) | 0.500752 / 0.579283 (-0.078531) | 0.620963 / 0.434364 (0.186599) | 0.607995 / 0.540337 (0.067658) | 0.722915 / 1.386936 (-0.664021) |

![](https://cml.dev/watermark.png#682d21e94ab1e64c11b583de39dc4c93f0101c5a "CML watermark")


Avoid stuck map operation when subprocesses crashes

Hi ! Do you think this can be fixed at the Pool level ? Ideally it should be the Pool responsibility to handle this, not the `map` code. We could even subclass Pool if needed (at least the one from `multiprocess`)

@lhoestq it makes sense to me. Just pushed a refactoring creating a `class ProcessPool(multiprocess.pool.Pool)` to keep track of the PID changes.

I managed to raise an error without subclassing Pool with two additions to `iflatmap_unordered`:

1. at the beggining
```python
original_pool = list(pool._pool)
```

2. in the loop
```python
if any(async_result._pool != original_pool for async_result in async_results) and queue.empty():
    raise RuntimeError(
        "One of the subprocesses has abruptly died during map operation."
        "To debug the error, disable multiprocessing."
    )
```

It's still a fix that only works for `iflatmap_unordered` (so not for map, imap etc) but is maybe simpler that subclassing. It also works for both multiprocessing.Pool and multiprocess.Pool

@lhoestq sorry for the delay. Busy weeks here.

I just pushed the change you requested. It looks closer to the original proposal, actually.

It seems that `map` actually uses `iflatmap_unordered` ([here](https://github.com/huggingface/datasets/blob/819bb4346434912eb405ce3f3e9f21dc25a2fe85/src/datasets/arrow_dataset.py#L1509)). I think this solution works fine for the `map` method (which is the one being tested by the new `tests/test_arrow_dataset.py::BaseDatasetTest::test_map_crash_subprocess`, right?).

It looks all good to me, feel free to fix code formatting by running `make style` and we can merge :)

> Yes fixing iflatmap_unordered does fix Dataset.map, but it won't fix any Pool.map that we may use elsewhere so we'll have to keep this in mind.

Right, I agree. The best way moving forward is probably not using the buggy `multiprocess.Pool` anymore, and replace it with `concurrent.futures.ProcessPoolExecutor` as much as possible.

Anyway, I've run `make style` now. Thanks for the support!

It looks like checking the async_result._pool doesn't always work - sorry about that. We might just go back to your original solution then. Would also be cool to open an issue in `multiprocess` to ask if they have a solution or if they plan to fix this.

@lhoestq no problem! Reverted to the previous version.

TBH, given the discussions [in this python issue](https://github.com/python/cpython/issues/66587), I don't think the error in `multiprocess` will be merged upstream any time soon...

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006060 / 0.011353 (-0.005293) | 0.003695 / 0.011008 (-0.007313) | 0.080484 / 0.038508 (0.041976) | 0.061894 / 0.023109 (0.038785) | 0.312510 / 0.275898 (0.036612) | 0.352398 / 0.323480 (0.028918) | 0.004638 / 0.007986 (-0.003348) | 0.002918 / 0.004328 (-0.001410) | 0.062932 / 0.004250 (0.058681) | 0.050859 / 0.037052 (0.013807) | 0.316812 / 0.258489 (0.058323) | 0.357684 / 0.293841 (0.063843) | 0.027622 / 0.128546 (-0.100924) | 0.008012 / 0.075646 (-0.067634) | 0.260970 / 0.419271 (-0.158302) | 0.045807 / 0.043533 (0.002275) | 0.321235 / 0.255139 (0.066096) | 0.343162 / 0.283200 (0.059962) | 0.021136 / 0.141683 (-0.120547) | 1.465886 / 1.452155 (0.013731) | 1.500216 / 1.492716 (0.007500) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.187286 / 0.018006 (0.169279) | 0.428724 / 0.000490 (0.428235) | 0.003029 / 0.000200 (0.002829) | 0.000063 / 0.000054 (0.000008) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022703 / 0.037411 (-0.014708) | 0.072740 / 0.014526 (0.058215) | 0.083436 / 0.176557 (-0.093120) | 0.144559 / 0.737135 (-0.592577) | 0.083958 / 0.296338 (-0.212380) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.435729 / 0.215209 (0.220520) | 4.351146 / 2.077655 (2.273491) | 2.316627 / 1.504120 (0.812508) | 2.144587 / 1.541195 (0.603393) | 2.209182 / 1.468490 (0.740692) | 0.501131 / 4.584777 (-4.083646) | 3.077085 / 3.745712 (-0.668627) | 4.353706 / 5.269862 (-0.916156) | 2.621523 / 4.565676 (-1.944154) | 0.058976 / 0.424275 (-0.365299) | 0.006467 / 0.007607 (-0.001141) | 0.506690 / 0.226044 (0.280646) | 5.085787 / 2.268929 (2.816858) | 2.731336 / 55.444624 (-52.713289) | 2.419451 / 6.876477 (-4.457025) | 2.583649 / 2.142072 (0.441577) | 0.589869 / 4.805227 (-4.215359) | 0.131040 / 6.500664 (-6.369624) | 0.061332 / 0.075469 (-0.014137) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.220542 / 1.841788 (-0.621245) | 18.169643 / 8.074308 (10.095335) | 13.251704 / 10.191392 (3.060312) | 0.142952 / 0.680424 (-0.537472) | 0.016639 / 0.534201 (-0.517562) | 0.334851 / 0.579283 (-0.244432) | 0.361865 / 0.434364 (-0.072499) | 0.380933 / 0.540337 (-0.159404) | 0.527374 / 1.386936 (-0.859562) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006319 / 0.011353 (-0.005034) | 0.003778 / 0.011008 (-0.007231) | 0.062388 / 0.038508 (0.023880) | 0.062228 / 0.023109 (0.039119) | 0.373727 / 0.275898 (0.097829) | 0.399442 / 0.323480 (0.075962) | 0.005434 / 0.007986 (-0.002551) | 0.003020 / 0.004328 (-0.001308) | 0.062774 / 0.004250 (0.058524) | 0.052784 / 0.037052 (0.015732) | 0.376428 / 0.258489 (0.117939) | 0.405039 / 0.293841 (0.111198) | 0.027884 / 0.128546 (-0.100662) | 0.008086 / 0.075646 (-0.067561) | 0.067078 / 0.419271 (-0.352194) | 0.042927 / 0.043533 (-0.000606) | 0.372142 / 0.255139 (0.117003) | 0.389604 / 0.283200 (0.106405) | 0.021582 / 0.141683 (-0.120101) | 1.473332 / 1.452155 (0.021177) | 1.536018 / 1.492716 (0.043302) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.184729 / 0.018006 (0.166723) | 0.421065 / 0.000490 (0.420575) | 0.002681 / 0.000200 (0.002481) | 0.000070 / 0.000054 (0.000015) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026067 / 0.037411 (-0.011344) | 0.077138 / 0.014526 (0.062612) | 0.085178 / 0.176557 (-0.091379) | 0.139681 / 0.737135 (-0.597454) | 0.087528 / 0.296338 (-0.208810) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.444899 / 0.215209 (0.229690) | 4.459168 / 2.077655 (2.381513) | 2.408792 / 1.504120 (0.904672) | 2.237243 / 1.541195 (0.696048) | 2.296298 / 1.468490 (0.827808) | 0.498508 / 4.584777 (-4.086269) | 3.067064 / 3.745712 (-0.678648) | 4.470577 / 5.269862 (-0.799284) | 2.701972 / 4.565676 (-1.863705) | 0.057711 / 0.424275 (-0.366564) | 0.006443 / 0.007607 (-0.001164) | 0.524046 / 0.226044 (0.298002) | 5.229928 / 2.268929 (2.961000) | 2.862101 / 55.444624 (-52.582523) | 2.545972 / 6.876477 (-4.330504) | 2.606459 / 2.142072 (0.464387) | 0.593285 / 4.805227 (-4.211942) | 0.124913 / 6.500664 (-6.375751) | 0.061942 / 0.075469 (-0.013527) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.322162 / 1.841788 (-0.519625) | 18.745796 / 8.074308 (10.671488) | 13.955443 / 10.191392 (3.764051) | 0.145610 / 0.680424 (-0.534814) | 0.016817 / 0.534201 (-0.517384) | 0.331180 / 0.579283 (-0.248103) | 0.343019 / 0.434364 (-0.091345) | 0.379459 / 0.540337 (-0.160878) | 0.526403 / 1.386936 (-0.860533) |

![](https://cml.dev/watermark.png#aca4cdcc79f16ec5157a2a3a665fdef0e3aa176d "CML watermark")

I've been using Dataset.map() with `num_proc=os.cpu_count()` to leverage multicore processing for my datasets, but from time to time I get stuck processes waiting forever. Apparently, when one of the subprocesses is abruptly killed (OOM killer, segfault, SIGKILL, etc), the main process keeps waiting for the async task sent to that child process to finish.

It seems to be easy to reproduce the issue with the following script:

```
import os
from datasets import Dataset, Features, Value

def do_stuck(item):
    os.kill(os.getpid(), 9)

data = {
    "col1": list(range(5)),
    "col2": list(range(5)),
}

ds = Dataset.from_dict(
    data,
    features=Features({
        "col1": Value("int64"),
        "col2": Value("int64"),
    }),
)

print(ds.map(do_stuck, num_proc=4))
```

This is an old behavior in Python, which apparently was fixed a few years ago in `concurrent.futures.ProcessPoolExecutor` ([ref](https://bugs.python.org/issue9205)), but not in `multiprocessing.pool.Pool` / `multiprocess.pool.Pool`, which is used by `Dataset.map` ([ref](https://bugs.python.org/issue22393)).

This PR is an idea to try to detect when a child process gets killed, and raises a `RuntimeError` warning the dataset.map() caller.

EDIT: Related proposal for future improvement: https://github.com/huggingface/datasets/discussions/5977


Streaming Dataset behind Proxy - FileNotFoundError

Hi ! can you try to set the upper case environment variables `HTTP_PROXY` and `HTTPS_PROXY` ?

We use `aiohttp` for streaming and it uses case sensitive environment variables

Hi, thanks for the quick reply.

I set the uppercase env variables with

`
os.environ['HTTP_PROXY'] = "http://example.com:xxxx"
os.environ['HTTPS_PROXY'] = "http://example.com:xxxx"
`

However, I still get the same error.

One thing that could be helpfull: When downloading a dataset without streaming i get the following message:
_HF google storage unreachable. Downloading and preparing it from source_.
The download does however work as expected.

Are you able to use `aiohttp` to get the file at `https://huggingface.co/datasets/facebook/voxpopuli/resolve/main/data/n_files.json` using your proxy ?

It only works when passing trust_env=True when creating the ClientSession, as well as setting ssl=False.

Working Example:

```
import os

os.environ['HTTP_PROXY'] = "xyz"
os.environ['HTTPS_PROXY'] = "xyz"

import asyncio
import aiohttp

async def download_pep(url):
    async with aiohttp.ClientSession(trust_env=True) as session:
        print("1")
        async with session.get(url, ssl=False) as resp:
            print("2")
            content = await resp.text()
            print(content)
            return content

asyncio.run(download_pep("https://huggingface.co/datasets/facebook/voxpopuli/resolve/main/data/n_files.json"))
```

SSL Verification has been a problem with other packages as well. Usually I circumvent the problem by setting
```
import ssl
ssl._create_default_https_context = ssl._create_unverified_context
```
(probably not the best idea for security), although here aiohttp does not seem to use this default context.

We do pass `trust_env` as well. Could you share the full stack trace you get when streaming using `datasets` ? That could help locate where we might have forgotten to pass `trust_env`

Is there a way to disable ssl verification when streaming a dataset. I suspect this might be the isssue with my proxy.

Here you go:

```
FileNotFoundError                         Traceback (most recent call last)
Cell In[8], line 3
      1 from datasets import load_dataset
----> 3 ds = load_dataset("facebook/voxpopuli", name="de", streaming=True)
      5 sample = next(iter(ds))

File [~/.conda/envs/audio_hf/lib/python3.10/site-packages/datasets/load.py:1790](https://vscode-remote+ssh-002dremote-002bml-002er-002dsoftware-002eat.vscode-resource.vscode-cdn.net/home/wrsbri/projects/audio_course/~/.conda/envs/audio_hf/lib/python3.10/site-packages/datasets/load.py:1790), in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, verification_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, num_proc, storage_options, **config_kwargs)
   1788 # Return iterable dataset in case of streaming
   1789 if streaming:
-> 1790     return builder_instance.as_streaming_dataset(split=split)
   1792 # Some datasets are already processed on the HF google storage
   1793 # Don't try downloading from Google storage for the packaged datasets as text, json, csv or pandas
   1794 try_from_hf_gcs = path not in _PACKAGED_DATASETS_MODULES

File [~/.conda/envs/audio_hf/lib/python3.10/site-packages/datasets/builder.py:1281](https://vscode-remote+ssh-002dremote-002bml-002er-002dsoftware-002eat.vscode-resource.vscode-cdn.net/home/wrsbri/projects/audio_course/~/.conda/envs/audio_hf/lib/python3.10/site-packages/datasets/builder.py:1281), in DatasetBuilder.as_streaming_dataset(self, split, base_path)
   1274 dl_manager = StreamingDownloadManager(
   1275     base_path=base_path or self.base_path,
   1276     download_config=DownloadConfig(use_auth_token=self.use_auth_token, storage_options=self.storage_options),
   1277     dataset_name=self.name,
   1278     data_dir=self.config.data_dir,
   1279 )
   1280 self._check_manual_download(dl_manager)
-> 1281 splits_generators = {sg.name: sg for sg in self._split_generators(dl_manager)}
   1282 # By default, return all splits
   1283 if split is None:

File [~/.cache/huggingface/modules/datasets_modules/datasets/facebook--voxpopuli/b5ff837284f0778eefe0f642734e142d8c3f574eba8c9c8a4b13602297f73604/voxpopuli.py:120](https://vscode-remote+ssh-002dremote-002bml-002er-002dsoftware-002eat.vscode-resource.vscode-cdn.net/home/wrsbri/projects/audio_course/~/.cache/huggingface/modules/datasets_modules/datasets/facebook--voxpopuli/b5ff837284f0778eefe0f642734e142d8c3f574eba8c9c8a4b13602297f73604/voxpopuli.py:120), in Voxpopuli._split_generators(self, dl_manager)
    118 def _split_generators(self, dl_manager):
    119     n_shards_path = dl_manager.download_and_extract(_N_SHARDS_FILE)
--> 120     with open(n_shards_path) as f:
    121         n_shards = json.load(f)
    123     if self.config.name == "en_accented":

File [~/.conda/envs/audio_hf/lib/python3.10/site-packages/datasets/streaming.py:71](https://vscode-remote+ssh-002dremote-002bml-002er-002dsoftware-002eat.vscode-resource.vscode-cdn.net/home/wrsbri/projects/audio_course/~/.conda/envs/audio_hf/lib/python3.10/site-packages/datasets/streaming.py:71), in extend_module_for_streaming..wrap_auth..wrapper(*args, **kwargs)
     69 @wraps(function)
     70 def wrapper(*args, **kwargs):
---> 71     return function(*args, use_auth_token=use_auth_token, **kwargs)

File [~/.conda/envs/audio_hf/lib/python3.10/site-packages/datasets/download/streaming_download_manager.py:517](https://vscode-remote+ssh-002dremote-002bml-002er-002dsoftware-002eat.vscode-resource.vscode-cdn.net/home/wrsbri/projects/audio_course/~/.conda/envs/audio_hf/lib/python3.10/site-packages/datasets/download/streaming_download_manager.py:517), in xopen(file, mode, use_auth_token, *args, **kwargs)
    515 except FileNotFoundError:
    516     if file.startswith(config.HF_ENDPOINT):
--> 517         raise FileNotFoundError(
    518             file + "\nIf the repo is private or gated, make sure to log in with `huggingface-cli login`."
    519         ) from None
    520     else:
    521         raise

FileNotFoundError: https://huggingface.co/datasets/facebook/voxpopuli/resolve/main/data/n_files.json
If the repo is private or gated, make sure to log in with `huggingface-cli login`.
```

> Is there a way to disable ssl verification when streaming a dataset.

I don't think so.

We use `fsspec` HTTPFileSystem implementation that is based on `aiohttp`. If you register a subclass of HTTPFileSystem that has SSL disabled by default it could work, but I wouldn't recommended it because it can raise security issues.

Okay thanks for your help! I guess I have to figure out how to improve the proxy environment / see if I can make it work with ssl connections.

### Describe the bug

When trying to stream a dataset i get the following error after a few minutes of waiting.

```
FileNotFoundError: https://huggingface.co/datasets/facebook/voxpopuli/resolve/main/data/n_files.json
If the repo is private or gated, make sure to log in with `huggingface-cli login`.
```

I have already set the proxy environment variables. Downloading a Dataset without streaming works as expected.
Still i suspect that this is connected to being behind a proxy.

Is there a way to set the proxy for streaming datasets? Possibly a keyword argument that gets passed to ffspec?

### Steps to reproduce the bug

This is the code i use.

```
import os
os.environ['http_proxy'] = "http://example.com:xxxx"
os.environ['https_proxy'] = "http://example.com:xxxx"

from datasets import load_dataset

ds = load_dataset("facebook/voxpopuli", name="de", streaming=True)
```

### Expected behavior

I would expect the streaming functionality to use the set proxy settings.

### Environment info

- `datasets` version: 2.13.0
- Platform: Linux-5.15.0-73-generic-x86_64-with-glibc2.35
- Python version: 3.10.11
- Huggingface_hub version: 0.15.1
- PyArrow version: 11.0.0
- Pandas version: 2.0.2


Deprecate `errors` param in favor of `encoding_errors` in text builder

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006518 / 0.011353 (-0.004835) | 0.004121 / 0.011008 (-0.006887) | 0.103350 / 0.038508 (0.064842) | 0.045030 / 0.023109 (0.021920) | 0.351670 / 0.275898 (0.075772) | 0.408110 / 0.323480 (0.084630) | 0.003883 / 0.007986 (-0.004102) | 0.003352 / 0.004328 (-0.000977) | 0.078786 / 0.004250 (0.074535) | 0.063977 / 0.037052 (0.026925) | 0.369759 / 0.258489 (0.111270) | 0.415103 / 0.293841 (0.121262) | 0.033069 / 0.128546 (-0.095477) | 0.008863 / 0.075646 (-0.066783) | 0.353660 / 0.419271 (-0.065611) | 0.055714 / 0.043533 (0.012181) | 0.350458 / 0.255139 (0.095319) | 0.369505 / 0.283200 (0.086305) | 0.022822 / 0.141683 (-0.118861) | 1.537588 / 1.452155 (0.085433) | 1.590569 / 1.492716 (0.097853) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.206826 / 0.018006 (0.188819) | 0.471625 / 0.000490 (0.471135) | 0.005188 / 0.000200 (0.004988) | 0.000316 / 0.000054 (0.000261) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028148 / 0.037411 (-0.009263) | 0.111941 / 0.014526 (0.097415) | 0.122106 / 0.176557 (-0.054451) | 0.181127 / 0.737135 (-0.556009) | 0.127534 / 0.296338 (-0.168805) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.409520 / 0.215209 (0.194311) | 4.098455 / 2.077655 (2.020800) | 1.852447 / 1.504120 (0.348327) | 1.657036 / 1.541195 (0.115842) | 1.709624 / 1.468490 (0.241134) | 0.542806 / 4.584777 (-4.041970) | 3.809352 / 3.745712 (0.063640) | 1.855412 / 5.269862 (-3.414449) | 1.109180 / 4.565676 (-3.456497) | 0.066801 / 0.424275 (-0.357474) | 0.011832 / 0.007607 (0.004225) | 0.518338 / 0.226044 (0.292293) | 5.190108 / 2.268929 (2.921179) | 2.320602 / 55.444624 (-53.124023) | 1.991416 / 6.876477 (-4.885060) | 2.106989 / 2.142072 (-0.035084) | 0.668914 / 4.805227 (-4.136313) | 0.145325 / 6.500664 (-6.355340) | 0.065145 / 0.075469 (-0.010324) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.254706 / 1.841788 (-0.587082) | 14.707264 / 8.074308 (6.632956) | 14.615423 / 10.191392 (4.424031) | 0.170764 / 0.680424 (-0.509659) | 0.017905 / 0.534201 (-0.516296) | 0.435606 / 0.579283 (-0.143677) | 0.434648 / 0.434364 (0.000284) | 0.520813 / 0.540337 (-0.019524) | 0.633902 / 1.386936 (-0.753034) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007212 / 0.011353 (-0.004141) | 0.004301 / 0.011008 (-0.006707) | 0.080767 / 0.038508 (0.042258) | 0.051949 / 0.023109 (0.028840) | 0.398473 / 0.275898 (0.122575) | 0.465038 / 0.323480 (0.141558) | 0.005580 / 0.007986 (-0.002406) | 0.003556 / 0.004328 (-0.000773) | 0.080682 / 0.004250 (0.076431) | 0.059517 / 0.037052 (0.022464) | 0.421171 / 0.258489 (0.162682) | 0.459752 / 0.293841 (0.165911) | 0.032960 / 0.128546 (-0.095586) | 0.009107 / 0.075646 (-0.066539) | 0.086382 / 0.419271 (-0.332889) | 0.056053 / 0.043533 (0.012520) | 0.393357 / 0.255139 (0.138218) | 0.412972 / 0.283200 (0.129772) | 0.031115 / 0.141683 (-0.110568) | 1.576961 / 1.452155 (0.124806) | 1.627249 / 1.492716 (0.134533) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.227618 / 0.018006 (0.209612) | 0.444640 / 0.000490 (0.444150) | 0.004376 / 0.000200 (0.004176) | 0.000092 / 0.000054 (0.000038) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030826 / 0.037411 (-0.006586) | 0.117587 / 0.014526 (0.103062) | 0.127467 / 0.176557 (-0.049089) | 0.184440 / 0.737135 (-0.552695) | 0.133664 / 0.296338 (-0.162675) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.443183 / 0.215209 (0.227974) | 4.408312 / 2.077655 (2.330658) | 2.132487 / 1.504120 (0.628367) | 1.923632 / 1.541195 (0.382438) | 1.967882 / 1.468490 (0.499392) | 0.552954 / 4.584777 (-4.031823) | 3.777701 / 3.745712 (0.031989) | 1.857686 / 5.269862 (-3.412176) | 1.104847 / 4.565676 (-3.460829) | 0.068350 / 0.424275 (-0.355925) | 0.012437 / 0.007607 (0.004830) | 0.559258 / 0.226044 (0.333214) | 5.593258 / 2.268929 (3.324330) | 2.648059 / 55.444624 (-52.796565) | 2.277428 / 6.876477 (-4.599049) | 2.351685 / 2.142072 (0.209612) | 0.678750 / 4.805227 (-4.126477) | 0.145550 / 6.500664 (-6.355114) | 0.066556 / 0.075469 (-0.008913) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.327128 / 1.841788 (-0.514659) | 15.649079 / 8.074308 (7.574771) | 14.478659 / 10.191392 (4.287267) | 0.147633 / 0.680424 (-0.532791) | 0.018502 / 0.534201 (-0.515699) | 0.438556 / 0.579283 (-0.140727) | 0.433381 / 0.434364 (-0.000983) | 0.514367 / 0.540337 (-0.025970) | 0.618347 / 1.386936 (-0.768589) |

![](https://cml.dev/watermark.png#16aa1c886c5b499641a4bb3d8ce4a4f7de8244b7 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006078 / 0.011353 (-0.005275) | 0.003914 / 0.011008 (-0.007095) | 0.102039 / 0.038508 (0.063531) | 0.037660 / 0.023109 (0.014551) | 0.348963 / 0.275898 (0.073065) | 0.407284 / 0.323480 (0.083804) | 0.004661 / 0.007986 (-0.003324) | 0.003253 / 0.004328 (-0.001076) | 0.078276 / 0.004250 (0.074025) | 0.054144 / 0.037052 (0.017091) | 0.376715 / 0.258489 (0.118225) | 0.418499 / 0.293841 (0.124658) | 0.027627 / 0.128546 (-0.100919) | 0.008494 / 0.075646 (-0.067152) | 0.316894 / 0.419271 (-0.102377) | 0.046560 / 0.043533 (0.003027) | 0.339835 / 0.255139 (0.084696) | 0.374628 / 0.283200 (0.091428) | 0.020729 / 0.141683 (-0.120954) | 1.502769 / 1.452155 (0.050615) | 1.548756 / 1.492716 (0.056040) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.229192 / 0.018006 (0.211186) | 0.426245 / 0.000490 (0.425756) | 0.005190 / 0.000200 (0.004990) | 0.000081 / 0.000054 (0.000026) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024271 / 0.037411 (-0.013140) | 0.098869 / 0.014526 (0.084343) | 0.105079 / 0.176557 (-0.071477) | 0.164707 / 0.737135 (-0.572428) | 0.110337 / 0.296338 (-0.186002) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.426593 / 0.215209 (0.211383) | 4.293977 / 2.077655 (2.216323) | 1.928502 / 1.504120 (0.424382) | 1.728623 / 1.541195 (0.187428) | 1.792084 / 1.468490 (0.323594) | 0.568737 / 4.584777 (-4.016040) | 3.438534 / 3.745712 (-0.307178) | 1.797798 / 5.269862 (-3.472063) | 1.054078 / 4.565676 (-3.511598) | 0.068711 / 0.424275 (-0.355564) | 0.011250 / 0.007607 (0.003643) | 0.529299 / 0.226044 (0.303255) | 5.283965 / 2.268929 (3.015037) | 2.358274 / 55.444624 (-53.086350) | 2.012818 / 6.876477 (-4.863659) | 2.109923 / 2.142072 (-0.032149) | 0.679556 / 4.805227 (-4.125671) | 0.138346 / 6.500664 (-6.362318) | 0.066349 / 0.075469 (-0.009120) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.193994 / 1.841788 (-0.647794) | 14.073158 / 8.074308 (5.998850) | 13.488525 / 10.191392 (3.297133) | 0.144536 / 0.680424 (-0.535888) | 0.016748 / 0.534201 (-0.517453) | 0.362703 / 0.579283 (-0.216580) | 0.389511 / 0.434364 (-0.044853) | 0.427296 / 0.540337 (-0.113041) | 0.513227 / 1.386936 (-0.873709) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006215 / 0.011353 (-0.005138) | 0.003834 / 0.011008 (-0.007174) | 0.078001 / 0.038508 (0.039493) | 0.036537 / 0.023109 (0.013428) | 0.369724 / 0.275898 (0.093826) | 0.426761 / 0.323480 (0.103281) | 0.003602 / 0.007986 (-0.004383) | 0.003001 / 0.004328 (-0.001327) | 0.075989 / 0.004250 (0.071739) | 0.048618 / 0.037052 (0.011566) | 0.374296 / 0.258489 (0.115807) | 0.430330 / 0.293841 (0.136489) | 0.028299 / 0.128546 (-0.100247) | 0.008537 / 0.075646 (-0.067109) | 0.083275 / 0.419271 (-0.335997) | 0.043136 / 0.043533 (-0.000397) | 0.359072 / 0.255139 (0.103933) | 0.387391 / 0.283200 (0.104192) | 0.021202 / 0.141683 (-0.120481) | 1.520832 / 1.452155 (0.068677) | 1.567030 / 1.492716 (0.074313) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.230944 / 0.018006 (0.212938) | 0.422159 / 0.000490 (0.421669) | 0.003447 / 0.000200 (0.003247) | 0.000125 / 0.000054 (0.000071) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025442 / 0.037411 (-0.011969) | 0.103944 / 0.014526 (0.089418) | 0.110577 / 0.176557 (-0.065979) | 0.161393 / 0.737135 (-0.575743) | 0.113482 / 0.296338 (-0.182857) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.485765 / 0.215209 (0.270556) | 4.845737 / 2.077655 (2.768083) | 2.556732 / 1.504120 (1.052612) | 2.348638 / 1.541195 (0.807443) | 2.379289 / 1.468490 (0.910799) | 0.561261 / 4.584777 (-4.023516) | 3.482468 / 3.745712 (-0.263244) | 3.061319 / 5.269862 (-2.208543) | 1.483938 / 4.565676 (-3.081738) | 0.067584 / 0.424275 (-0.356691) | 0.011333 / 0.007607 (0.003726) | 0.594342 / 0.226044 (0.368297) | 5.935477 / 2.268929 (3.666548) | 3.025029 / 55.444624 (-52.419595) | 2.687032 / 6.876477 (-4.189445) | 2.752470 / 2.142072 (0.610398) | 0.674470 / 4.805227 (-4.130757) | 0.136777 / 6.500664 (-6.363887) | 0.068335 / 0.075469 (-0.007134) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.336456 / 1.841788 (-0.505332) | 14.376007 / 8.074308 (6.301699) | 14.171375 / 10.191392 (3.979983) | 0.159620 / 0.680424 (-0.520804) | 0.016685 / 0.534201 (-0.517516) | 0.364344 / 0.579283 (-0.214939) | 0.395358 / 0.434364 (-0.039006) | 0.424876 / 0.540337 (-0.115461) | 0.513267 / 1.386936 (-0.873669) |

![](https://cml.dev/watermark.png#6ed837325cb539a5deb99129e5ad181d0269e050 "CML watermark")

For consistency with the JSON builder and Pandas