Filter unsupported extensions

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006983 / 0.011353 (-0.004369) | 0.004473 / 0.011008 (-0.006535) | 0.105158 / 0.038508 (0.066650) | 0.048973 / 0.023109 (0.025864) | 0.358771 / 0.275898 (0.082873) | 0.432389 / 0.323480 (0.108909) | 0.005689 / 0.007986 (-0.002297) | 0.003584 / 0.004328 (-0.000744) | 0.080852 / 0.004250 (0.076601) | 0.066133 / 0.037052 (0.029081) | 0.370981 / 0.258489 (0.112492) | 0.406942 / 0.293841 (0.113101) | 0.032123 / 0.128546 (-0.096424) | 0.009313 / 0.075646 (-0.066333) | 0.355220 / 0.419271 (-0.064051) | 0.055768 / 0.043533 (0.012235) | 0.370545 / 0.255139 (0.115406) | 0.375619 / 0.283200 (0.092419) | 0.024258 / 0.141683 (-0.117425) | 1.559073 / 1.452155 (0.106918) | 1.616520 / 1.492716 (0.123804) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.277893 / 0.018006 (0.259887) | 0.535447 / 0.000490 (0.534957) | 0.004877 / 0.000200 (0.004677) | 0.000092 / 0.000054 (0.000037) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029444 / 0.037411 (-0.007968) | 0.114366 / 0.014526 (0.099841) | 0.130957 / 0.176557 (-0.045599) | 0.189604 / 0.737135 (-0.547531) | 0.131682 / 0.296338 (-0.164656) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.412315 / 0.215209 (0.197106) | 4.093879 / 2.077655 (2.016225) | 1.856169 / 1.504120 (0.352050) | 1.655358 / 1.541195 (0.114164) | 1.758190 / 1.468490 (0.289699) | 0.545829 / 4.584777 (-4.038948) | 3.871436 / 3.745712 (0.125724) | 1.938244 / 5.269862 (-3.331618) | 1.122727 / 4.565676 (-3.442950) | 0.067107 / 0.424275 (-0.357168) | 0.012012 / 0.007607 (0.004405) | 0.518868 / 0.226044 (0.292824) | 5.235081 / 2.268929 (2.966153) | 2.335115 / 55.444624 (-53.109509) | 2.013074 / 6.876477 (-4.863402) | 2.219808 / 2.142072 (0.077735) | 0.674602 / 4.805227 (-4.130626) | 0.147051 / 6.500664 (-6.353613) | 0.068444 / 0.075469 (-0.007025) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.245600 / 1.841788 (-0.596188) | 15.537727 / 8.074308 (7.463419) | 15.074300 / 10.191392 (4.882908) | 0.194217 / 0.680424 (-0.486207) | 0.018536 / 0.534201 (-0.515665) | 0.437085 / 0.579283 (-0.142198) | 0.441123 / 0.434364 (0.006759) | 0.530681 / 0.540337 (-0.009657) | 0.649154 / 1.386936 (-0.737782) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007243 / 0.011353 (-0.004110) | 0.004688 / 0.011008 (-0.006320) | 0.079809 / 0.038508 (0.041301) | 0.046915 / 0.023109 (0.023805) | 0.415144 / 0.275898 (0.139246) | 0.474867 / 0.323480 (0.151388) | 0.004550 / 0.007986 (-0.003435) | 0.004585 / 0.004328 (0.000257) | 0.080837 / 0.004250 (0.076587) | 0.061667 / 0.037052 (0.024614) | 0.411321 / 0.258489 (0.152832) | 0.464195 / 0.293841 (0.170354) | 0.032510 / 0.128546 (-0.096037) | 0.009306 / 0.075646 (-0.066340) | 0.086637 / 0.419271 (-0.332635) | 0.053335 / 0.043533 (0.009802) | 0.402302 / 0.255139 (0.147163) | 0.424864 / 0.283200 (0.141664) | 0.026573 / 0.141683 (-0.115110) | 1.566793 / 1.452155 (0.114639) | 1.628118 / 1.492716 (0.135401) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.317802 / 0.018006 (0.299796) | 0.544593 / 0.000490 (0.544103) | 0.005690 / 0.000200 (0.005490) | 0.000107 / 0.000054 (0.000053) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033015 / 0.037411 (-0.004397) | 0.121940 / 0.014526 (0.107414) | 0.132920 / 0.176557 (-0.043637) | 0.191481 / 0.737135 (-0.545655) | 0.139139 / 0.296338 (-0.157199) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.460382 / 0.215209 (0.245173) | 4.610046 / 2.077655 (2.532392) | 2.296573 / 1.504120 (0.792453) | 2.099735 / 1.541195 (0.558540) | 2.213913 / 1.468490 (0.745423) | 0.544871 / 4.584777 (-4.039906) | 3.814174 / 3.745712 (0.068462) | 3.246397 / 5.269862 (-2.023464) | 1.480236 / 4.565676 (-3.085440) | 0.068464 / 0.424275 (-0.355811) | 0.012651 / 0.007607 (0.005043) | 0.564989 / 0.226044 (0.338944) | 5.639188 / 2.268929 (3.370259) | 2.827601 / 55.444624 (-52.617023) | 2.473743 / 6.876477 (-4.402734) | 2.567413 / 2.142072 (0.425340) | 0.674351 / 4.805227 (-4.130876) | 0.146248 / 6.500664 (-6.354416) | 0.067553 / 0.075469 (-0.007916) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.346703 / 1.841788 (-0.495085) | 16.494787 / 8.074308 (8.420479) | 15.179487 / 10.191392 (4.988095) | 0.181864 / 0.680424 (-0.498560) | 0.018857 / 0.534201 (-0.515344) | 0.437787 / 0.579283 (-0.141496) | 0.431770 / 0.434364 (-0.002594) | 0.507116 / 0.540337 (-0.033221) | 0.608899 / 1.386936 (-0.778037) |

![](https://cml.dev/watermark.png#0fd5b7412f907675e76b183a6e39ef6d176fdcc0 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005963 / 0.011353 (-0.005390) | 0.003743 / 0.011008 (-0.007265) | 0.098519 / 0.038508 (0.060011) | 0.037392 / 0.023109 (0.014283) | 0.322706 / 0.275898 (0.046808) | 0.380032 / 0.323480 (0.056552) | 0.004694 / 0.007986 (-0.003292) | 0.002897 / 0.004328 (-0.001432) | 0.078664 / 0.004250 (0.074414) | 0.052646 / 0.037052 (0.015594) | 0.335523 / 0.258489 (0.077034) | 0.375464 / 0.293841 (0.081623) | 0.027537 / 0.128546 (-0.101010) | 0.008452 / 0.075646 (-0.067194) | 0.313844 / 0.419271 (-0.105427) | 0.047368 / 0.043533 (0.003835) | 0.313833 / 0.255139 (0.058694) | 0.342284 / 0.283200 (0.059085) | 0.021136 / 0.141683 (-0.120547) | 1.544764 / 1.452155 (0.092610) | 1.563850 / 1.492716 (0.071134) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.188609 / 0.018006 (0.170603) | 0.421686 / 0.000490 (0.421196) | 0.003336 / 0.000200 (0.003136) | 0.000077 / 0.000054 (0.000023) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023678 / 0.037411 (-0.013733) | 0.099191 / 0.014526 (0.084665) | 0.105819 / 0.176557 (-0.070738) | 0.169654 / 0.737135 (-0.567481) | 0.110240 / 0.296338 (-0.186099) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.425497 / 0.215209 (0.210288) | 4.237165 / 2.077655 (2.159510) | 1.902953 / 1.504120 (0.398833) | 1.699012 / 1.541195 (0.157818) | 1.751107 / 1.468490 (0.282617) | 0.563326 / 4.584777 (-4.021451) | 3.394189 / 3.745712 (-0.351523) | 2.706129 / 5.269862 (-2.563732) | 1.361522 / 4.565676 (-3.204155) | 0.067776 / 0.424275 (-0.356499) | 0.010959 / 0.007607 (0.003352) | 0.530905 / 0.226044 (0.304860) | 5.322467 / 2.268929 (3.053538) | 2.384356 / 55.444624 (-53.060269) | 2.044196 / 6.876477 (-4.832281) | 2.119837 / 2.142072 (-0.022235) | 0.682236 / 4.805227 (-4.122991) | 0.136921 / 6.500664 (-6.363743) | 0.066784 / 0.075469 (-0.008685) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.210642 / 1.841788 (-0.631146) | 13.804572 / 8.074308 (5.730264) | 13.309229 / 10.191392 (3.117837) | 0.154356 / 0.680424 (-0.526068) | 0.016833 / 0.534201 (-0.517368) | 0.366503 / 0.579283 (-0.212780) | 0.385201 / 0.434364 (-0.049163) | 0.426713 / 0.540337 (-0.113624) | 0.516795 / 1.386936 (-0.870141) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006144 / 0.011353 (-0.005209) | 0.003723 / 0.011008 (-0.007285) | 0.077427 / 0.038508 (0.038919) | 0.037636 / 0.023109 (0.014527) | 0.375048 / 0.275898 (0.099150) | 0.442254 / 0.323480 (0.118774) | 0.003506 / 0.007986 (-0.004480) | 0.003751 / 0.004328 (-0.000577) | 0.076771 / 0.004250 (0.072521) | 0.047915 / 0.037052 (0.010862) | 0.378918 / 0.258489 (0.120429) | 0.435300 / 0.293841 (0.141459) | 0.028317 / 0.128546 (-0.100230) | 0.008413 / 0.075646 (-0.067233) | 0.082774 / 0.419271 (-0.336497) | 0.043211 / 0.043533 (-0.000321) | 0.362022 / 0.255139 (0.106883) | 0.404928 / 0.283200 (0.121728) | 0.020692 / 0.141683 (-0.120991) | 1.527303 / 1.452155 (0.075148) | 1.596091 / 1.492716 (0.103375) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.225537 / 0.018006 (0.207530) | 0.399901 / 0.000490 (0.399412) | 0.000424 / 0.000200 (0.000224) | 0.000058 / 0.000054 (0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026483 / 0.037411 (-0.010928) | 0.104373 / 0.014526 (0.089847) | 0.111271 / 0.176557 (-0.065286) | 0.163872 / 0.737135 (-0.573264) | 0.113991 / 0.296338 (-0.182347) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.456484 / 0.215209 (0.241275) | 4.572652 / 2.077655 (2.494998) | 2.374908 / 1.504120 (0.870788) | 2.207855 / 1.541195 (0.666661) | 2.260009 / 1.468490 (0.791519) | 0.562678 / 4.584777 (-4.022099) | 3.441778 / 3.745712 (-0.303934) | 1.729006 / 5.269862 (-3.540855) | 1.024937 / 4.565676 (-3.540739) | 0.068707 / 0.424275 (-0.355568) | 0.011334 / 0.007607 (0.003727) | 0.564293 / 0.226044 (0.338248) | 5.638367 / 2.268929 (3.369438) | 2.665654 / 55.444624 (-52.778970) | 2.320033 / 6.876477 (-4.556444) | 2.328706 / 2.142072 (0.186634) | 0.677433 / 4.805227 (-4.127794) | 0.137190 / 6.500664 (-6.363474) | 0.068585 / 0.075469 (-0.006885) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.312476 / 1.841788 (-0.529312) | 14.206685 / 8.074308 (6.132377) | 14.217928 / 10.191392 (4.026536) | 0.143416 / 0.680424 (-0.537007) | 0.016647 / 0.534201 (-0.517554) | 0.361228 / 0.579283 (-0.218055) | 0.396185 / 0.434364 (-0.038178) | 0.423275 / 0.540337 (-0.117063) | 0.512966 / 1.386936 (-0.873970) |

![](https://cml.dev/watermark.png#b424648fd68bd0b5279eb916cec4836d1220e268 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008913 / 0.011353 (-0.002440) | 0.005142 / 0.011008 (-0.005866) | 0.133958 / 0.038508 (0.095449) | 0.049180 / 0.023109 (0.026071) | 0.389169 / 0.275898 (0.113270) | 0.481513 / 0.323480 (0.158033) | 0.006555 / 0.007986 (-0.001430) | 0.003806 / 0.004328 (-0.000522) | 0.102056 / 0.004250 (0.097806) | 0.083259 / 0.037052 (0.046207) | 0.392536 / 0.258489 (0.134047) | 0.447503 / 0.293841 (0.153662) | 0.047472 / 0.128546 (-0.081074) | 0.014748 / 0.075646 (-0.060899) | 0.475619 / 0.419271 (0.056348) | 0.107306 / 0.043533 (0.063773) | 0.421942 / 0.255139 (0.166803) | 0.419736 / 0.283200 (0.136536) | 0.044195 / 0.141683 (-0.097488) | 1.793840 / 1.452155 (0.341686) | 1.960204 / 1.492716 (0.467488) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.252046 / 0.018006 (0.234040) | 0.627725 / 0.000490 (0.627236) | 0.007435 / 0.000200 (0.007235) | 0.000526 / 0.000054 (0.000472) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034656 / 0.037411 (-0.002755) | 0.114534 / 0.014526 (0.100008) | 0.135804 / 0.176557 (-0.040753) | 0.209309 / 0.737135 (-0.527826) | 0.140369 / 0.296338 (-0.155969) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.636736 / 0.215209 (0.421527) | 6.039985 / 2.077655 (3.962330) | 2.640141 / 1.504120 (1.136021) | 2.284492 / 1.541195 (0.743297) | 2.324956 / 1.468490 (0.856466) | 0.934499 / 4.584777 (-3.650278) | 5.673415 / 3.745712 (1.927703) | 5.184584 / 5.269862 (-0.085278) | 2.661911 / 4.565676 (-1.903766) | 0.150420 / 0.424275 (-0.273855) | 0.015655 / 0.007607 (0.008048) | 0.748290 / 0.226044 (0.522246) | 7.579755 / 2.268929 (5.310827) | 3.346732 / 55.444624 (-52.097892) | 2.708212 / 6.876477 (-4.168264) | 2.682423 / 2.142072 (0.540351) | 1.170389 / 4.805227 (-3.634838) | 0.215775 / 6.500664 (-6.284889) | 0.076360 / 0.075469 (0.000891) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.516794 / 1.841788 (-0.324993) | 18.709117 / 8.074308 (10.634809) | 22.492542 / 10.191392 (12.301150) | 0.237978 / 0.680424 (-0.442446) | 0.027828 / 0.534201 (-0.506373) | 0.499968 / 0.579283 (-0.079315) | 0.645899 / 0.434364 (0.211535) | 0.548599 / 0.540337 (0.008262) | 0.675428 / 1.386936 (-0.711508) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008469 / 0.011353 (-0.002884) | 0.005420 / 0.011008 (-0.005589) | 0.093340 / 0.038508 (0.054832) | 0.045896 / 0.023109 (0.022786) | 0.533267 / 0.275898 (0.257369) | 0.596034 / 0.323480 (0.272555) | 0.004816 / 0.007986 (-0.003170) | 0.004379 / 0.004328 (0.000051) | 0.096356 / 0.004250 (0.092106) | 0.058339 / 0.037052 (0.021287) | 0.574464 / 0.258489 (0.315975) | 0.649301 / 0.293841 (0.355461) | 0.047599 / 0.128546 (-0.080947) | 0.013759 / 0.075646 (-0.061887) | 0.104672 / 0.419271 (-0.314599) | 0.061658 / 0.043533 (0.018125) | 0.560956 / 0.255139 (0.305817) | 0.585328 / 0.283200 (0.302128) | 0.034137 / 0.141683 (-0.107546) | 1.844528 / 1.452155 (0.392373) | 1.971398 / 1.492716 (0.478682) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.278666 / 0.018006 (0.260660) | 0.577342 / 0.000490 (0.576853) | 0.005496 / 0.000200 (0.005296) | 0.000131 / 0.000054 (0.000076) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029863 / 0.037411 (-0.007549) | 0.161703 / 0.014526 (0.147177) | 0.132279 / 0.176557 (-0.044277) | 0.227345 / 0.737135 (-0.509791) | 0.138047 / 0.296338 (-0.158291) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.651535 / 0.215209 (0.436326) | 7.077949 / 2.077655 (5.000295) | 2.926990 / 1.504120 (1.422871) | 2.598872 / 1.541195 (1.057678) | 2.614192 / 1.468490 (1.145702) | 0.913845 / 4.584777 (-3.670932) | 5.704301 / 3.745712 (1.958589) | 2.796914 / 5.269862 (-2.472948) | 1.836096 / 4.565676 (-2.729580) | 0.106294 / 0.424275 (-0.317981) | 0.012705 / 0.007607 (0.005098) | 0.836336 / 0.226044 (0.610291) | 8.234079 / 2.268929 (5.965150) | 3.836410 / 55.444624 (-51.608215) | 3.116752 / 6.876477 (-3.759724) | 3.154258 / 2.142072 (1.012186) | 1.195794 / 4.805227 (-3.609434) | 0.240491 / 6.500664 (-6.260173) | 0.087913 / 0.075469 (0.012444) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.724723 / 1.841788 (-0.117064) | 19.492194 / 8.074308 (11.417885) | 21.443341 / 10.191392 (11.251949) | 0.245819 / 0.680424 (-0.434605) | 0.027024 / 0.534201 (-0.507177) | 0.481071 / 0.579283 (-0.098212) | 0.596359 / 0.434364 (0.161995) | 0.646462 / 0.540337 (0.106124) | 0.706380 / 1.386936 (-0.680556) |

![](https://cml.dev/watermark.png#67ca664e6d5ef137127b238aae1d0aff54e22db2 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006634 / 0.011353 (-0.004719) | 0.004003 / 0.011008 (-0.007005) | 0.097874 / 0.038508 (0.059365) | 0.043528 / 0.023109 (0.020419) | 0.302293 / 0.275898 (0.026395) | 0.357041 / 0.323480 (0.033561) | 0.003761 / 0.007986 (-0.004225) | 0.004312 / 0.004328 (-0.000016) | 0.076253 / 0.004250 (0.072003) | 0.062807 / 0.037052 (0.025755) | 0.316737 / 0.258489 (0.058248) | 0.356722 / 0.293841 (0.062881) | 0.030816 / 0.128546 (-0.097730) | 0.008691 / 0.075646 (-0.066955) | 0.328366 / 0.419271 (-0.090906) | 0.062299 / 0.043533 (0.018766) | 0.293877 / 0.255139 (0.038738) | 0.319832 / 0.283200 (0.036632) | 0.024996 / 0.141683 (-0.116687) | 1.473912 / 1.452155 (0.021758) | 1.565439 / 1.492716 (0.072723) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.208428 / 0.018006 (0.190422) | 0.435618 / 0.000490 (0.435128) | 0.000695 / 0.000200 (0.000495) | 0.000056 / 0.000054 (0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026253 / 0.037411 (-0.011158) | 0.106908 / 0.014526 (0.092382) | 0.117075 / 0.176557 (-0.059482) | 0.177969 / 0.737135 (-0.559166) | 0.123400 / 0.296338 (-0.172938) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.424970 / 0.215209 (0.209761) | 4.203233 / 2.077655 (2.125578) | 2.009679 / 1.504120 (0.505559) | 1.825691 / 1.541195 (0.284496) | 1.870639 / 1.468490 (0.402149) | 0.530758 / 4.584777 (-4.054019) | 3.718791 / 3.745712 (-0.026921) | 1.800206 / 5.269862 (-3.469656) | 1.071651 / 4.565676 (-3.494025) | 0.065126 / 0.424275 (-0.359149) | 0.011312 / 0.007607 (0.003704) | 0.532503 / 0.226044 (0.306458) | 5.353950 / 2.268929 (3.085021) | 2.463548 / 55.444624 (-52.981076) | 2.139832 / 6.876477 (-4.736645) | 2.238722 / 2.142072 (0.096650) | 0.655736 / 4.805227 (-4.149492) | 0.141689 / 6.500664 (-6.358975) | 0.063282 / 0.075469 (-0.012187) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.183523 / 1.841788 (-0.658265) | 14.146428 / 8.074308 (6.072120) | 14.312883 / 10.191392 (4.121491) | 0.169286 / 0.680424 (-0.511138) | 0.017343 / 0.534201 (-0.516858) | 0.397934 / 0.579283 (-0.181349) | 0.417791 / 0.434364 (-0.016573) | 0.463639 / 0.540337 (-0.076698) | 0.562787 / 1.386936 (-0.824149) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006594 / 0.011353 (-0.004759) | 0.004086 / 0.011008 (-0.006922) | 0.075122 / 0.038508 (0.036614) | 0.041849 / 0.023109 (0.018740) | 0.362645 / 0.275898 (0.086747) | 0.464350 / 0.323480 (0.140870) | 0.003760 / 0.007986 (-0.004226) | 0.003327 / 0.004328 (-0.001001) | 0.076154 / 0.004250 (0.071904) | 0.053232 / 0.037052 (0.016180) | 0.407863 / 0.258489 (0.149374) | 0.460787 / 0.293841 (0.166946) | 0.031917 / 0.128546 (-0.096630) | 0.008770 / 0.075646 (-0.066876) | 0.082612 / 0.419271 (-0.336660) | 0.051311 / 0.043533 (0.007779) | 0.354508 / 0.255139 (0.099369) | 0.419533 / 0.283200 (0.136334) | 0.023980 / 0.141683 (-0.117703) | 1.491255 / 1.452155 (0.039100) | 1.536101 / 1.492716 (0.043384) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.178261 / 0.018006 (0.160255) | 0.444680 / 0.000490 (0.444190) | 0.013761 / 0.000200 (0.013561) | 0.000117 / 0.000054 (0.000063) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027875 / 0.037411 (-0.009536) | 0.111269 / 0.014526 (0.096744) | 0.121096 / 0.176557 (-0.055461) | 0.174387 / 0.737135 (-0.562749) | 0.124714 / 0.296338 (-0.171624) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.445422 / 0.215209 (0.230213) | 4.435877 / 2.077655 (2.358222) | 2.221895 / 1.504120 (0.717775) | 2.030571 / 1.541195 (0.489376) | 2.074863 / 1.468490 (0.606373) | 0.543331 / 4.584777 (-4.041446) | 3.753615 / 3.745712 (0.007903) | 3.317074 / 5.269862 (-1.952787) | 1.630390 / 4.565676 (-2.935286) | 0.066726 / 0.424275 (-0.357549) | 0.011556 / 0.007607 (0.003949) | 0.546985 / 0.226044 (0.320941) | 5.460634 / 2.268929 (3.191705) | 2.705945 / 55.444624 (-52.738679) | 2.373425 / 6.876477 (-4.503052) | 2.401472 / 2.142072 (0.259399) | 0.663225 / 4.805227 (-4.142002) | 0.143694 / 6.500664 (-6.356970) | 0.065283 / 0.075469 (-0.010186) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.264804 / 1.841788 (-0.576983) | 14.803228 / 8.074308 (6.728919) | 14.178514 / 10.191392 (3.987122) | 0.162651 / 0.680424 (-0.517772) | 0.017586 / 0.534201 (-0.516615) | 0.398740 / 0.579283 (-0.180543) | 0.414478 / 0.434364 (-0.019886) | 0.465442 / 0.540337 (-0.074895) | 0.563450 / 1.386936 (-0.823486) |

![](https://cml.dev/watermark.png#76f75a9a3b2aaad05ea0ea5ab77e01fd2ca66760 "CML watermark")

I used a regex to filter the data files based on their extension for packaged builders.

I tried and a regex is 10x faster that using `in` to check if the extension is in the list of supported extensions.

Supersedes https://github.com/huggingface/datasets/pull/5850

Close https://github.com/huggingface/datasets/issues/5849

I also did a small change to favor the parquet module in case of a draw in the extension counter.


Docs: make "repository structure" easier to find

documentation

Improvements or additions to documentation

Loading a local dataset also works the same way when `data_files` are not specified, so I agree we should make this info  easier to discover

cc @stevhliu

Is this issue open? If so, I will self assign.

@benjaminbrown038 Yes, it is. Maybe @stevhliu can give some pointers on improving this doc page's discoverability.

I think we can add a version of the [Main use-case](https://huggingface.co/docs/datasets/repository_structure#main-usecase) section to the [Share a dataset to the Hub](https://huggingface.co/docs/datasets/upload_dataset) tutorial.

Currently, it doesn't tell you *how* to structure the repository; it only tells you how to create it. So adding the "main use-case" will help bridge the gap and make it easier to find. We should also add a link to the [Structure your repository](https://huggingface.co/docs/datasets/repository_structure) guide for users who want to learn about the other options.

The page https://huggingface.co/docs/datasets/repository_structure explains how to create a simple repository structure without a dataset script.
It's the simplest way to create a dataset and should be easier to find, particularly on the docs' first pages.


description disappearing from Info when Uploading a Dataset Created with `from_dict`

Here's a minimal way to reproduce the bug, for the sake of convenience.
````
from datasets import Dataset, DatasetInfo, load_dataset

episodes_dict = {"test":[1,2,3],"test2": [1,2,4]}

hugging_face_dataset = Dataset.from_dict(
    episodes_dict, info=DatasetInfo(description="test_str")
)
print(hugging_face_dataset.info)

hugging_face_dataset.push_to_hub("balisujohn/minari_test", private=True)

redownloaded_dataset= load_dataset("balisujohn/minari_test")["train"]

print(redownloaded_dataset.info)
````

Thanks for reporting !

For now I would recommend uploading a separate JSON file for your metadata.

Alternatively you can upload a second configuration of the dataset containing your metadata but this feature is not released yet (though you can already use it from [here](https://github.com/huggingface/datasets/pull/5331), it will be released soon)

### Describe the bug

When uploading a dataset created locally using `from_dict` with a specified `description` field. It appears before upload, but is missing after upload and re-download.

### Steps to reproduce the bug

I think the most relevant pattern in the code might be the following lines:

```
description_json_str = json.dumps(
    {
        "dataset_id": dataset.spec.dataset_id,
        "env_name": dataset.spec.env_spec.id,
        "action_space": serialize_space(dataset.spec.action_space),
        "observation_space": serialize_space(dataset.spec.observation_space),
    }
)

hugging_face_dataset = Dataset.from_dict(
    episodes_dict, info=DatasetInfo(description=description_json_str)
)

```
Which comes from this function https://github.com/balisujohn/minarai/blob/8e023727f0a8488c4451651d9f7a79b981412c40/minari/integrations/hugging_face.py#L39

To replicate,
clone this branch of my Minari fork https://github.com/balisujohn/minarai/tree/dev-huggingface then run

```
python3.8 -m venv env
source env/bin/activate
python3 -m pip install -e .
python3 -m pip install pytest
```

The change the hugging face repo path in the test called  `test_hugging_face_push_and_pull_dataset` in `tests/integrations/test_hugging_face.py` to one you have permissions to write to.

Then run:

```
pytest tests/integrations/test_hugging_face.py::test_hugging_face_push_and_pull_dataset
```

### Expected behavior

DATASET INFO BEFORE UPLOADING
DatasetInfo(description='{"dataset_id": "dummy-combo-test-v0", "env_name": "DummyComboEnv-v0", "action_space": "{\\"type\\": \\"Tuple\\", \\"subspaces\\": [{\\"type\\": \\"Box\\", \\"dtype\\": \\"float32\\", \\"shape\\": [1], \\"low\\": [2.0], \\"high\\": [3.0]}, {\\"type\\": \\"Box\\", \\"dtype\\": \\"float32\\", \\"shape\\": [1], \\"low\\": [4.0], \\"high\\": [5.0]}]}", "observation_space": "{\\"type\\": \\"Tuple\\", \\"subspaces\\": [{\\"type\\": \\"Box\\", \\"dtype\\": \\"float32\\", \\"shape\\": [1], \\"low\\": [2.0], \\"high\\": [3.0]}, {\\"type\\": \\"Tuple\\", \\"subspaces\\": [{\\"type\\": \\"Box\\", \\"dtype\\": \\"float32\\", \\"shape\\": [1], \\"low\\": [2.0], \\"high\\": [3.0]}, {\\"type\\": \\"Dict\\", \\"subspaces\\": {\\"component_1\\": {\\"type\\": \\"Box\\", \\"dtype\\": \\"float32\\", \\"shape\\": [1], \\"low\\": [-1.0], \\"high\\": [1.0]}, \\"component_2\\": {\\"type\\": \\"Dict\\", \\"subspaces\\": {\\"subcomponent_1\\": {\\"type\\": \\"Box\\", \\"dtype\\": \\"float32\\", \\"shape\\": [1], \\"low\\": [2.0], \\"high\\": [3.0]}, \\"subcomponent_2\\": {\\"type\\": \\"Tuple\\", \\"subspaces\\": [{\\"type\\": \\"Box\\", \\"dtype\\": \\"float32\\", \\"shape\\": [1], \\"low\\": [4.0], \\"high\\": [5.0]}, {\\"type\\": \\"Discrete\\", \\"dtype\\": \\"int64\\", \\"start\\": 0, \\"n\\": 10}]}}}}}]}]}"}', citation='', homepage='', license='', features={'observations': {'_index_0': Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), '_index_1': {'_index_0': Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), '_index_1': {'component_1': Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), 'component_2': {'subcomponent_1': Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), 'subcomponent_2': {'_index_0': Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), '_index_1': Value(dtype='int64', id=None)}}}}}, 'actions': {'_index_0': Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), '_index_1': Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None)}, 'rewards': Value(dtype='int64', id=None), 'truncations': Value(dtype='bool', id=None), 'terminations': Value(dtype='bool', id=None), 'episode_ids': Value(dtype='int64', id=None)}, post_processed=None, supervised_keys=None, task_templates=None, builder_name=None, config_name=None, version=None, splits=None, download_checksums=None, download_size=None, post_processing_size=None, dataset_size=None, size_in_bytes=None)
...
DATASET INFO AFTER UPLOADING AND DOWNLOADING
DatasetInfo(description='', citation='', homepage='', license='', features={'observations': {'_index_0': Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), '_index_1': {'_index_0': Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), '_index_1': {'component_1': Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), 'component_2': {'subcomponent_1': Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), 'subcomponent_2': {'_index_0': Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), '_index_1': Value(dtype='int64', id=None)}}}}}, 'actions': {'_index_0': Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), '_index_1': Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None)}, 'rewards': Value(dtype='int64', id=None), 'truncations': Value(dtype='bool', id=None), 'terminations': Value(dtype='bool', id=None), 'episode_ids': Value(dtype='int64', id=None)}, post_processed=None, supervised_keys=None, task_templates=None, builder_name=None, config_name=None, version=None, splits={'train': SplitInfo(name='train', num_bytes=4846, num_examples=60, shard_lengths=None, dataset_name='parquet')}, download_checksums={'https://huggingface.co/datasets/balisujohn/minari_test/resolve/8217b614ff9ba5edc1a30c7df430e92a46f65363/data/train-00000-of-00001-7c5900b93b35745e.parquet': {'num_bytes': 9052, 'checksum': None}}, download_size=9052, post_processing_size=None, dataset_size=4846, size_in_bytes=13898)
...

### Environment info

- `datasets` version: 2.13.0
- Platform: Linux-5.15.0-75-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.1
- Pandas version: 2.0.2


Add `encoding` and `errors` params to JSON loader

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006770 / 0.011353 (-0.004583) | 0.004143 / 0.011008 (-0.006865) | 0.098928 / 0.038508 (0.060420) | 0.044893 / 0.023109 (0.021783) | 0.302630 / 0.275898 (0.026732) | 0.368173 / 0.323480 (0.044693) | 0.005631 / 0.007986 (-0.002354) | 0.003397 / 0.004328 (-0.000931) | 0.075748 / 0.004250 (0.071497) | 0.062582 / 0.037052 (0.025530) | 0.329586 / 0.258489 (0.071097) | 0.362625 / 0.293841 (0.068784) | 0.033250 / 0.128546 (-0.095296) | 0.008880 / 0.075646 (-0.066766) | 0.329683 / 0.419271 (-0.089588) | 0.054426 / 0.043533 (0.010893) | 0.297940 / 0.255139 (0.042801) | 0.319796 / 0.283200 (0.036597) | 0.023296 / 0.141683 (-0.118387) | 1.462142 / 1.452155 (0.009987) | 1.495796 / 1.492716 (0.003079) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.201771 / 0.018006 (0.183765) | 0.454514 / 0.000490 (0.454024) | 0.003333 / 0.000200 (0.003133) | 0.000081 / 0.000054 (0.000027) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028084 / 0.037411 (-0.009327) | 0.109452 / 0.014526 (0.094926) | 0.119200 / 0.176557 (-0.057357) | 0.180302 / 0.737135 (-0.556834) | 0.125653 / 0.296338 (-0.170686) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.409819 / 0.215209 (0.194610) | 4.055117 / 2.077655 (1.977462) | 1.855279 / 1.504120 (0.351159) | 1.655281 / 1.541195 (0.114086) | 1.687938 / 1.468490 (0.219448) | 0.528352 / 4.584777 (-4.056425) | 3.750250 / 3.745712 (0.004538) | 3.386741 / 5.269862 (-1.883121) | 1.572036 / 4.565676 (-2.993640) | 0.065125 / 0.424275 (-0.359150) | 0.011259 / 0.007607 (0.003652) | 0.513449 / 0.226044 (0.287405) | 5.139421 / 2.268929 (2.870492) | 2.316973 / 55.444624 (-53.127651) | 1.984109 / 6.876477 (-4.892368) | 2.127915 / 2.142072 (-0.014158) | 0.653238 / 4.805227 (-4.151989) | 0.142686 / 6.500664 (-6.357978) | 0.063666 / 0.075469 (-0.011803) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.185174 / 1.841788 (-0.656614) | 14.790282 / 8.074308 (6.715974) | 13.089222 / 10.191392 (2.897830) | 0.146055 / 0.680424 (-0.534369) | 0.017835 / 0.534201 (-0.516366) | 0.399598 / 0.579283 (-0.179685) | 0.425296 / 0.434364 (-0.009068) | 0.478552 / 0.540337 (-0.061786) | 0.579702 / 1.386936 (-0.807234) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006750 / 0.011353 (-0.004603) | 0.004156 / 0.011008 (-0.006853) | 0.074948 / 0.038508 (0.036440) | 0.043368 / 0.023109 (0.020259) | 0.355389 / 0.275898 (0.079491) | 0.429167 / 0.323480 (0.105687) | 0.003911 / 0.007986 (-0.004075) | 0.004340 / 0.004328 (0.000012) | 0.075940 / 0.004250 (0.071689) | 0.054293 / 0.037052 (0.017241) | 0.400317 / 0.258489 (0.141827) | 0.432001 / 0.293841 (0.138160) | 0.032340 / 0.128546 (-0.096206) | 0.008876 / 0.075646 (-0.066770) | 0.082284 / 0.419271 (-0.336987) | 0.050819 / 0.043533 (0.007286) | 0.351994 / 0.255139 (0.096855) | 0.375917 / 0.283200 (0.092717) | 0.022466 / 0.141683 (-0.119217) | 1.538824 / 1.452155 (0.086669) | 1.563995 / 1.492716 (0.071279) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.227330 / 0.018006 (0.209323) | 0.446380 / 0.000490 (0.445890) | 0.000408 / 0.000200 (0.000208) | 0.000058 / 0.000054 (0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028534 / 0.037411 (-0.008878) | 0.113467 / 0.014526 (0.098941) | 0.123590 / 0.176557 (-0.052966) | 0.174309 / 0.737135 (-0.562827) | 0.130631 / 0.296338 (-0.165707) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.441020 / 0.215209 (0.225811) | 4.386564 / 2.077655 (2.308909) | 2.100704 / 1.504120 (0.596584) | 1.901484 / 1.541195 (0.360289) | 1.963494 / 1.468490 (0.495004) | 0.536838 / 4.584777 (-4.047939) | 3.739071 / 3.745712 (-0.006642) | 3.278981 / 5.269862 (-1.990881) | 1.515476 / 4.565676 (-3.050201) | 0.066388 / 0.424275 (-0.357887) | 0.011857 / 0.007607 (0.004250) | 0.545507 / 0.226044 (0.319463) | 5.441479 / 2.268929 (3.172550) | 2.602144 / 55.444624 (-52.842480) | 2.235583 / 6.876477 (-4.640894) | 2.293458 / 2.142072 (0.151385) | 0.658535 / 4.805227 (-4.146692) | 0.141327 / 6.500664 (-6.359337) | 0.063726 / 0.075469 (-0.011743) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.247819 / 1.841788 (-0.593968) | 15.234524 / 8.074308 (7.160216) | 14.592700 / 10.191392 (4.401308) | 0.141952 / 0.680424 (-0.538472) | 0.017747 / 0.534201 (-0.516454) | 0.396819 / 0.579283 (-0.182465) | 0.415902 / 0.434364 (-0.018462) | 0.464619 / 0.540337 (-0.075718) | 0.560866 / 1.386936 (-0.826070) |

![](https://cml.dev/watermark.png#4b7f6c59deb868e21f295917548fa2df10dd0158 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008278 / 0.011353 (-0.003075) | 0.005044 / 0.011008 (-0.005964) | 0.123382 / 0.038508 (0.084874) | 0.054039 / 0.023109 (0.030929) | 0.382338 / 0.275898 (0.106440) | 0.453287 / 0.323480 (0.129807) | 0.006342 / 0.007986 (-0.001644) | 0.003930 / 0.004328 (-0.000398) | 0.094039 / 0.004250 (0.089789) | 0.076525 / 0.037052 (0.039472) | 0.394066 / 0.258489 (0.135577) | 0.445600 / 0.293841 (0.151759) | 0.039348 / 0.128546 (-0.089199) | 0.010485 / 0.075646 (-0.065161) | 0.433730 / 0.419271 (0.014459) | 0.082671 / 0.043533 (0.039138) | 0.375250 / 0.255139 (0.120111) | 0.416269 / 0.283200 (0.133070) | 0.038397 / 0.141683 (-0.103286) | 1.864834 / 1.452155 (0.412680) | 2.010453 / 1.492716 (0.517737) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.240008 / 0.018006 (0.222002) | 0.470975 / 0.000490 (0.470485) | 0.004001 / 0.000200 (0.003801) | 0.000097 / 0.000054 (0.000042) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031107 / 0.037411 (-0.006304) | 0.129371 / 0.014526 (0.114846) | 0.141559 / 0.176557 (-0.034997) | 0.205571 / 0.737135 (-0.531564) | 0.144611 / 0.296338 (-0.151728) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.506972 / 0.215209 (0.291763) | 5.055951 / 2.077655 (2.978296) | 2.397438 / 1.504120 (0.893318) | 2.170435 / 1.541195 (0.629240) | 2.240296 / 1.468490 (0.771806) | 0.641559 / 4.584777 (-3.943218) | 4.644772 / 3.745712 (0.899060) | 4.064200 / 5.269862 (-1.205662) | 1.946991 / 4.565676 (-2.618685) | 0.086413 / 0.424275 (-0.337862) | 0.015082 / 0.007607 (0.007475) | 0.670413 / 0.226044 (0.444369) | 6.331346 / 2.268929 (4.062418) | 2.965813 / 55.444624 (-52.478812) | 2.547952 / 6.876477 (-4.328524) | 2.718390 / 2.142072 (0.576318) | 0.796657 / 4.805227 (-4.008571) | 0.173229 / 6.500664 (-6.327435) | 0.079606 / 0.075469 (0.004137) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.568761 / 1.841788 (-0.273026) | 18.485432 / 8.074308 (10.411124) | 15.758513 / 10.191392 (5.567121) | 0.170427 / 0.680424 (-0.509997) | 0.021421 / 0.534201 (-0.512780) | 0.518623 / 0.579283 (-0.060660) | 0.525887 / 0.434364 (0.091523) | 0.640331 / 0.540337 (0.099993) | 0.766748 / 1.386936 (-0.620188) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007680 / 0.011353 (-0.003673) | 0.005289 / 0.011008 (-0.005719) | 0.093773 / 0.038508 (0.055265) | 0.054997 / 0.023109 (0.031888) | 0.456277 / 0.275898 (0.180379) | 0.500642 / 0.323480 (0.177162) | 0.005935 / 0.007986 (-0.002050) | 0.004375 / 0.004328 (0.000047) | 0.094131 / 0.004250 (0.089881) | 0.063399 / 0.037052 (0.026347) | 0.470546 / 0.258489 (0.212057) | 0.504989 / 0.293841 (0.211148) | 0.038541 / 0.128546 (-0.090006) | 0.010403 / 0.075646 (-0.065244) | 0.102469 / 0.419271 (-0.316802) | 0.063105 / 0.043533 (0.019572) | 0.466005 / 0.255139 (0.210866) | 0.458677 / 0.283200 (0.175477) | 0.028407 / 0.141683 (-0.113276) | 1.893829 / 1.452155 (0.441675) | 1.917954 / 1.492716 (0.425238) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.272760 / 0.018006 (0.254754) | 0.476159 / 0.000490 (0.475669) | 0.008467 / 0.000200 (0.008267) | 0.000146 / 0.000054 (0.000091) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.035755 / 0.037411 (-0.001656) | 0.145038 / 0.014526 (0.130512) | 0.148322 / 0.176557 (-0.028235) | 0.210193 / 0.737135 (-0.526943) | 0.156547 / 0.296338 (-0.139792) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.541204 / 0.215209 (0.325995) | 5.382746 / 2.077655 (3.305091) | 2.704229 / 1.504120 (1.200109) | 2.468422 / 1.541195 (0.927227) | 2.522672 / 1.468490 (1.054182) | 0.644899 / 4.584777 (-3.939878) | 4.654401 / 3.745712 (0.908689) | 2.159223 / 5.269862 (-3.110638) | 1.280098 / 4.565676 (-3.285578) | 0.080053 / 0.424275 (-0.344222) | 0.014383 / 0.007607 (0.006776) | 0.662770 / 0.226044 (0.436725) | 6.617651 / 2.268929 (4.348722) | 3.234347 / 55.444624 (-52.210277) | 2.861417 / 6.876477 (-4.015059) | 2.888928 / 2.142072 (0.746856) | 0.792854 / 4.805227 (-4.012374) | 0.172553 / 6.500664 (-6.328111) | 0.078402 / 0.075469 (0.002933) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.565351 / 1.841788 (-0.276436) | 18.681916 / 8.074308 (10.607608) | 17.264473 / 10.191392 (7.073081) | 0.168461 / 0.680424 (-0.511963) | 0.021353 / 0.534201 (-0.512848) | 0.517843 / 0.579283 (-0.061440) | 0.519907 / 0.434364 (0.085543) | 0.623687 / 0.540337 (0.083350) | 0.761796 / 1.386936 (-0.625140) |

![](https://cml.dev/watermark.png#bbf58747f734a46e75937bdbcbc05b06ade0224a "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006750 / 0.011353 (-0.004603) | 0.004268 / 0.011008 (-0.006741) | 0.098644 / 0.038508 (0.060136) | 0.044643 / 0.023109 (0.021534) | 0.309420 / 0.275898 (0.033522) | 0.379294 / 0.323480 (0.055815) | 0.005729 / 0.007986 (-0.002256) | 0.003615 / 0.004328 (-0.000714) | 0.076086 / 0.004250 (0.071835) | 0.068994 / 0.037052 (0.031942) | 0.325653 / 0.258489 (0.067164) | 0.375187 / 0.293841 (0.081347) | 0.032546 / 0.128546 (-0.096000) | 0.009089 / 0.075646 (-0.066557) | 0.329905 / 0.419271 (-0.089366) | 0.066832 / 0.043533 (0.023300) | 0.299247 / 0.255139 (0.044108) | 0.323460 / 0.283200 (0.040260) | 0.034226 / 0.141683 (-0.107457) | 1.475659 / 1.452155 (0.023505) | 1.556234 / 1.492716 (0.063518) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.292305 / 0.018006 (0.274299) | 0.542584 / 0.000490 (0.542094) | 0.003047 / 0.000200 (0.002847) | 0.000082 / 0.000054 (0.000027) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030096 / 0.037411 (-0.007315) | 0.112341 / 0.014526 (0.097815) | 0.124965 / 0.176557 (-0.051591) | 0.183159 / 0.737135 (-0.553976) | 0.131885 / 0.296338 (-0.164453) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.426437 / 0.215209 (0.211228) | 4.260984 / 2.077655 (2.183330) | 2.078358 / 1.504120 (0.574238) | 1.877644 / 1.541195 (0.336449) | 2.044036 / 1.468490 (0.575546) | 0.532980 / 4.584777 (-4.051797) | 3.749573 / 3.745712 (0.003860) | 1.944155 / 5.269862 (-3.325706) | 1.090307 / 4.565676 (-3.475370) | 0.065445 / 0.424275 (-0.358830) | 0.011237 / 0.007607 (0.003630) | 0.521448 / 0.226044 (0.295403) | 5.213118 / 2.268929 (2.944189) | 2.507829 / 55.444624 (-52.936795) | 2.177179 / 6.876477 (-4.699297) | 2.351161 / 2.142072 (0.209088) | 0.656775 / 4.805227 (-4.148452) | 0.141207 / 6.500664 (-6.359457) | 0.063286 / 0.075469 (-0.012183) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.190281 / 1.841788 (-0.651506) | 15.327424 / 8.074308 (7.253116) | 13.300695 / 10.191392 (3.109303) | 0.190484 / 0.680424 (-0.489939) | 0.017984 / 0.534201 (-0.516217) | 0.405714 / 0.579283 (-0.173569) | 0.435915 / 0.434364 (0.001551) | 0.494083 / 0.540337 (-0.046254) | 0.600616 / 1.386936 (-0.786320) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006740 / 0.011353 (-0.004613) | 0.004289 / 0.011008 (-0.006719) | 0.076532 / 0.038508 (0.038024) | 0.043305 / 0.023109 (0.020196) | 0.356111 / 0.275898 (0.080213) | 0.434121 / 0.323480 (0.110641) | 0.005599 / 0.007986 (-0.002387) | 0.003461 / 0.004328 (-0.000868) | 0.077097 / 0.004250 (0.072847) | 0.055369 / 0.037052 (0.018317) | 0.367093 / 0.258489 (0.108604) | 0.418801 / 0.293841 (0.124960) | 0.032057 / 0.128546 (-0.096489) | 0.009048 / 0.075646 (-0.066599) | 0.082897 / 0.419271 (-0.336374) | 0.050287 / 0.043533 (0.006754) | 0.352060 / 0.255139 (0.096921) | 0.376278 / 0.283200 (0.093078) | 0.023924 / 0.141683 (-0.117759) | 1.522780 / 1.452155 (0.070626) | 1.578938 / 1.492716 (0.086222) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.287317 / 0.018006 (0.269311) | 0.508490 / 0.000490 (0.508000) | 0.000431 / 0.000200 (0.000231) | 0.000056 / 0.000054 (0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031139 / 0.037411 (-0.006272) | 0.113927 / 0.014526 (0.099401) | 0.128147 / 0.176557 (-0.048409) | 0.179712 / 0.737135 (-0.557424) | 0.134364 / 0.296338 (-0.161975) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.452834 / 0.215209 (0.237625) | 4.507944 / 2.077655 (2.430289) | 2.287758 / 1.504120 (0.783638) | 2.091145 / 1.541195 (0.549951) | 2.196228 / 1.468490 (0.727738) | 0.539306 / 4.584777 (-4.045471) | 3.838941 / 3.745712 (0.093228) | 1.908801 / 5.269862 (-3.361060) | 1.139235 / 4.565676 (-3.426442) | 0.066677 / 0.424275 (-0.357599) | 0.011422 / 0.007607 (0.003815) | 0.562966 / 0.226044 (0.336921) | 5.633712 / 2.268929 (3.364784) | 2.788622 / 55.444624 (-52.656002) | 2.438465 / 6.876477 (-4.438012) | 2.523479 / 2.142072 (0.381407) | 0.668730 / 4.805227 (-4.136498) | 0.143977 / 6.500664 (-6.356687) | 0.064661 / 0.075469 (-0.010808) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.291708 / 1.841788 (-0.550080) | 15.573316 / 8.074308 (7.499008) | 14.435099 / 10.191392 (4.243707) | 0.147745 / 0.680424 (-0.532679) | 0.017602 / 0.534201 (-0.516599) | 0.401560 / 0.579283 (-0.177723) | 0.429861 / 0.434364 (-0.004502) | 0.469800 / 0.540337 (-0.070538) | 0.567515 / 1.386936 (-0.819421) |

![](https://cml.dev/watermark.png#79c340f5dcfd06340f180f6c6ea2d5ef81f49d98 "CML watermark")

"Requested" in https://discuss.huggingface.co/t/utf-16-for-datasets/43828/3.

`pd.read_json` also has these parameters, so it makes sense to be consistent.


Common Voice datasets still need `use_auth_token=True`

cc @pcuenca as well.

Not super urgent btw

The issue commes from the dataset itself and is not related to the `datasets` lib

see https://huggingface.co/datasets/mozilla-foundation/common_voice_6_1/blob/2c475b3b88e0f2e5828f830a4b91618a25ff20b7/common_voice_6_1.py#L148-L152

Let's remove these lines in the dataset no? cc @anton-l @Vaibhavs10

### Describe the bug

We don't need to pass `use_auth_token=True` anymore to download gated datasets or models, so the following should work if correctly logged in.

```py
from datasets import load_dataset

load_dataset("mozilla-foundation/common_voice_6_1", "tr", split="train+validation")
```

However it throws an error - probably because something weird is hardcoded into the dataset loading script.

### Steps to reproduce the bug

1.)
```
huggingface-cli login
```

2.) Make sure that you have accepted the license here:
https://huggingface.co/datasets/mozilla-foundation/common_voice_6_1

3.) Run:
```py
from datasets import load_dataset

load_dataset("mozilla-foundation/common_voice_6_1", "tr", split="train+validation")
```

4.)  You'll get:

```
File ~/hf/lib/python3.10/site-packages/datasets/builder.py:963, in DatasetBuilder._download_and_prepare(self, dl_manager, verification_mode, **prepare_split_kwargs)
    961 split_dict = SplitDict(dataset_name=self.name)
    962 split_generators_kwargs = self._make_split_generators_kwargs(prepare_split_kwargs)
--> 963 split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
    965 # Checksums verification
    966 if verification_mode == VerificationMode.ALL_CHECKS and dl_manager.record_checksums:

File ~/.cache/huggingface/modules/datasets_modules/datasets/mozilla-foundation--common_voice_6_1/f4d7854c466f5bd4908988dbd39044ec4fc634d89e0515ab0c51715c0127ffe3/common_voice_6_1.py:150, in CommonVoice._split_generators(self, dl_manager)
    148 hf_auth_token = dl_manager.download_config.use_auth_token
    149 if hf_auth_token is None:
--> 150     raise ConnectionError(
    151         "Please set use_auth_token=True or use_auth_token='<TOKEN>' to download this dataset"
    152     )
    154 bundle_url_template = STATS["bundleURLTemplate"]
    155 bundle_version = bundle_url_template.split("/")[0]

ConnectionError: Please set use_auth_token=True or use_auth_token='<TOKEN>' to download this dataset
```

### Expected behavior

One should not have to pass `use_auth_token=True`. Also see discussion here: https://github.com/huggingface/blog/pull/1243#discussion_r1235131150

### Environment info

```
- `datasets` version: 2.13.0
- Platform: Linux-6.2.0-76060200-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.16.0.dev0
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
```


Config name / split name lost after map with multiproc

This must be due to DatasetInfo.from_merge which drops them and is used in `concatenate_datasets`.

And you're experiencing this issue because multiprocessing does concatenate the resulting datasets from each process.

Maybe they should be kept if all the subdatasets share the same values for config_name and split

### Describe the bug

Performing a `.map` method on a dataset loses it's config name / split name only if run with multiproc

### Steps to reproduce the bug

```python
from datasets import Audio, load_dataset
from transformers import AutoFeatureExtractor
import numpy as np

# load dummy dataset
libri = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean")

# make train / test splits
libri = libri["validation"].train_test_split(seed=42, shuffle=True, test_size=0.1)

# example feature extractor
model_id = "ntu-spml/distilhubert"
feature_extractor = AutoFeatureExtractor.from_pretrained(model_id, do_normalize=True, return_attention_mask=True)

sampling_rate = feature_extractor.sampling_rate

libri = libri.cast_column("audio", Audio(sampling_rate=sampling_rate))

max_duration = 30.0

def preprocess_function(examples):
    audio_arrays = [x["array"] for x in examples["audio"]]
    inputs = feature_extractor(
        audio_arrays,
        sampling_rate=feature_extractor.sampling_rate,
        max_length=int(feature_extractor.sampling_rate * max_duration),
        truncation=True,
        return_attention_mask=True,
    )
    return inputs

# single proc map
libri_encoded = libri.map(
    preprocess_function, remove_columns=["audio", "file"], batched=True, num_proc=1
)

print(10 * "=" ,"Single processing", 10 * "=")
print("Config name before: ", libri["train"].config_name, " Split name before: ", libri["train"].split)
print("Config name after: ", libri_encoded["train"].config_name, " Split name after: ", libri_encoded["train"].split)

# multi proc map
libri_encoded = libri.map(
    preprocess_function, remove_columns=["audio", "file"], batched=True, num_proc=2
)

print(10 * "=" ,"Multi processing", 10 * "=")
print("Config name before: ", libri["train"].config_name, " Split name before: ", libri["train"].split)
print("Config name after: ", libri_encoded["train"].config_name, " Split name after: ", libri_encoded["train"].split)
```

**Print Output:**
```
========== Single processing ==========
Config name before:  clean  Split name before:  validation
Config name after:  clean  Split name after:  validation
========== Multi processing ==========
Config name before:  clean  Split name before:  validation
Config name after:  None  Split name after:  None
```

=> we can see that the config/split names are lost in the multiprocessing setting

### Expected behavior

Should retain both config / split names in the multiproc setting

### Environment info

- `datasets` version: 2.13.1.dev0
- Platform: Linux-5.15.0-67-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.2


Fix JSON generation in benchmarks CI

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006186 / 0.011353 (-0.005167) | 0.003744 / 0.011008 (-0.007264) | 0.097295 / 0.038508 (0.058787) | 0.037106 / 0.023109 (0.013997) | 0.424154 / 0.275898 (0.148256) | 0.474536 / 0.323480 (0.151057) | 0.003454 / 0.007986 (-0.004532) | 0.003865 / 0.004328 (-0.000463) | 0.077348 / 0.004250 (0.073097) | 0.051728 / 0.037052 (0.014675) | 0.437120 / 0.258489 (0.178631) | 0.478379 / 0.293841 (0.184538) | 0.028939 / 0.128546 (-0.099608) | 0.008376 / 0.075646 (-0.067270) | 0.312002 / 0.419271 (-0.107270) | 0.053723 / 0.043533 (0.010190) | 0.424815 / 0.255139 (0.169676) | 0.446203 / 0.283200 (0.163004) | 0.026553 / 0.141683 (-0.115130) | 1.479983 / 1.452155 (0.027828) | 1.530613 / 1.492716 (0.037896) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.196627 / 0.018006 (0.178620) | 0.422361 / 0.000490 (0.421871) | 0.003442 / 0.000200 (0.003242) | 0.000077 / 0.000054 (0.000023) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022913 / 0.037411 (-0.014499) | 0.096011 / 0.014526 (0.081485) | 0.104091 / 0.176557 (-0.072466) | 0.163273 / 0.737135 (-0.573862) | 0.109142 / 0.296338 (-0.187197) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.431032 / 0.215209 (0.215823) | 4.314391 / 2.077655 (2.236737) | 2.003812 / 1.504120 (0.499692) | 1.799538 / 1.541195 (0.258344) | 1.830026 / 1.468490 (0.361536) | 0.560131 / 4.584777 (-4.024646) | 3.368997 / 3.745712 (-0.376715) | 1.703032 / 5.269862 (-3.566830) | 1.026949 / 4.565676 (-3.538727) | 0.067507 / 0.424275 (-0.356768) | 0.010910 / 0.007607 (0.003303) | 0.532606 / 0.226044 (0.306562) | 5.345179 / 2.268929 (3.076250) | 2.368077 / 55.444624 (-53.076548) | 2.028913 / 6.876477 (-4.847564) | 2.147621 / 2.142072 (0.005549) | 0.675696 / 4.805227 (-4.129531) | 0.134902 / 6.500664 (-6.365762) | 0.065004 / 0.075469 (-0.010465) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.233412 / 1.841788 (-0.608376) | 13.767465 / 8.074308 (5.693157) | 13.933653 / 10.191392 (3.742261) | 0.129010 / 0.680424 (-0.551414) | 0.016708 / 0.534201 (-0.517493) | 0.362341 / 0.579283 (-0.216942) | 0.390902 / 0.434364 (-0.043462) | 0.429156 / 0.540337 (-0.111182) | 0.521166 / 1.386936 (-0.865770) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006169 / 0.011353 (-0.005184) | 0.003839 / 0.011008 (-0.007169) | 0.078784 / 0.038508 (0.040276) | 0.040218 / 0.023109 (0.017109) | 0.360439 / 0.275898 (0.084541) | 0.423957 / 0.323480 (0.100477) | 0.003456 / 0.007986 (-0.004529) | 0.002900 / 0.004328 (-0.001428) | 0.078820 / 0.004250 (0.074569) | 0.047240 / 0.037052 (0.010187) | 0.372081 / 0.258489 (0.113592) | 0.424263 / 0.293841 (0.130422) | 0.027977 / 0.128546 (-0.100569) | 0.008400 / 0.075646 (-0.067246) | 0.084399 / 0.419271 (-0.334872) | 0.043303 / 0.043533 (-0.000230) | 0.361583 / 0.255139 (0.106444) | 0.394987 / 0.283200 (0.111787) | 0.020006 / 0.141683 (-0.121677) | 1.520208 / 1.452155 (0.068053) | 1.587335 / 1.492716 (0.094619) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.223847 / 0.018006 (0.205840) | 0.402194 / 0.000490 (0.401704) | 0.000384 / 0.000200 (0.000184) | 0.000057 / 0.000054 (0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024902 / 0.037411 (-0.012509) | 0.099076 / 0.014526 (0.084550) | 0.108041 / 0.176557 (-0.068516) | 0.159385 / 0.737135 (-0.577750) | 0.111442 / 0.296338 (-0.184896) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.446232 / 0.215209 (0.231023) | 4.464927 / 2.077655 (2.387272) | 2.155234 / 1.504120 (0.651114) | 1.953645 / 1.541195 (0.412450) | 1.965991 / 1.468490 (0.497501) | 0.553473 / 4.584777 (-4.031304) | 3.321397 / 3.745712 (-0.424315) | 1.693761 / 5.269862 (-3.576101) | 1.006299 / 4.565676 (-3.559378) | 0.067013 / 0.424275 (-0.357262) | 0.011116 / 0.007607 (0.003509) | 0.555014 / 0.226044 (0.328970) | 5.535694 / 2.268929 (3.266765) | 2.598339 / 55.444624 (-52.846285) | 2.249298 / 6.876477 (-4.627179) | 2.243419 / 2.142072 (0.101347) | 0.667603 / 4.805227 (-4.137624) | 0.133322 / 6.500664 (-6.367343) | 0.065473 / 0.075469 (-0.009996) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.293051 / 1.841788 (-0.548737) | 14.103731 / 8.074308 (6.029423) | 14.215204 / 10.191392 (4.023812) | 0.143990 / 0.680424 (-0.536434) | 0.016805 / 0.534201 (-0.517396) | 0.363264 / 0.579283 (-0.216019) | 0.392769 / 0.434364 (-0.041594) | 0.425291 / 0.540337 (-0.115046) | 0.515479 / 1.386936 (-0.871457) |

![](https://cml.dev/watermark.png#e03a58f3f5d7e6f07279fb833e62d859a0babaad "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006346 / 0.011353 (-0.005006) | 0.004130 / 0.011008 (-0.006878) | 0.096898 / 0.038508 (0.058390) | 0.042564 / 0.023109 (0.019455) | 0.343748 / 0.275898 (0.067850) | 0.412515 / 0.323480 (0.089035) | 0.006153 / 0.007986 (-0.001833) | 0.003345 / 0.004328 (-0.000984) | 0.075314 / 0.004250 (0.071064) | 0.061478 / 0.037052 (0.024426) | 0.362948 / 0.258489 (0.104459) | 0.401533 / 0.293841 (0.107692) | 0.032363 / 0.128546 (-0.096184) | 0.008780 / 0.075646 (-0.066867) | 0.328691 / 0.419271 (-0.090580) | 0.054253 / 0.043533 (0.010721) | 0.340783 / 0.255139 (0.085644) | 0.360705 / 0.283200 (0.077505) | 0.023183 / 0.141683 (-0.118500) | 1.484078 / 1.452155 (0.031924) | 1.528581 / 1.492716 (0.035865) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.208732 / 0.018006 (0.190726) | 0.452572 / 0.000490 (0.452082) | 0.002936 / 0.000200 (0.002737) | 0.000082 / 0.000054 (0.000028) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024616 / 0.037411 (-0.012795) | 0.107547 / 0.014526 (0.093021) | 0.114492 / 0.176557 (-0.062065) | 0.171770 / 0.737135 (-0.565365) | 0.122538 / 0.296338 (-0.173800) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.406140 / 0.215209 (0.190930) | 4.062391 / 2.077655 (1.984736) | 1.865962 / 1.504120 (0.361842) | 1.682236 / 1.541195 (0.141041) | 1.738119 / 1.468490 (0.269629) | 0.532244 / 4.584777 (-4.052533) | 3.816421 / 3.745712 (0.070709) | 2.981205 / 5.269862 (-2.288656) | 1.519497 / 4.565676 (-3.046179) | 0.065904 / 0.424275 (-0.358371) | 0.011277 / 0.007607 (0.003670) | 0.512789 / 0.226044 (0.286745) | 5.107618 / 2.268929 (2.838690) | 2.419399 / 55.444624 (-53.025226) | 2.079262 / 6.876477 (-4.797214) | 2.150447 / 2.142072 (0.008375) | 0.696737 / 4.805227 (-4.108490) | 0.142497 / 6.500664 (-6.358167) | 0.063521 / 0.075469 (-0.011949) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.180692 / 1.841788 (-0.661095) | 14.343084 / 8.074308 (6.268776) | 13.303719 / 10.191392 (3.112327) | 0.164234 / 0.680424 (-0.516190) | 0.017439 / 0.534201 (-0.516762) | 0.399712 / 0.579283 (-0.179571) | 0.428248 / 0.434364 (-0.006115) | 0.471909 / 0.540337 (-0.068428) | 0.573853 / 1.386936 (-0.813083) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006210 / 0.011353 (-0.005143) | 0.004104 / 0.011008 (-0.006905) | 0.075140 / 0.038508 (0.036632) | 0.044647 / 0.023109 (0.021538) | 0.370120 / 0.275898 (0.094222) | 0.452936 / 0.323480 (0.129457) | 0.003943 / 0.007986 (-0.004042) | 0.003285 / 0.004328 (-0.001043) | 0.075267 / 0.004250 (0.071017) | 0.055517 / 0.037052 (0.018465) | 0.396385 / 0.258489 (0.137896) | 0.447870 / 0.293841 (0.154029) | 0.031342 / 0.128546 (-0.097204) | 0.008720 / 0.075646 (-0.066926) | 0.082702 / 0.419271 (-0.336570) | 0.051010 / 0.043533 (0.007477) | 0.350546 / 0.255139 (0.095407) | 0.425395 / 0.283200 (0.142195) | 0.024483 / 0.141683 (-0.117200) | 1.467341 / 1.452155 (0.015186) | 1.537187 / 1.492716 (0.044471) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.218067 / 0.018006 (0.200061) | 0.441603 / 0.000490 (0.441114) | 0.003711 / 0.000200 (0.003512) | 0.000092 / 0.000054 (0.000038) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028669 / 0.037411 (-0.008742) | 0.112941 / 0.014526 (0.098415) | 0.122584 / 0.176557 (-0.053972) | 0.176494 / 0.737135 (-0.560641) | 0.129369 / 0.296338 (-0.166970) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.434543 / 0.215209 (0.219334) | 4.344056 / 2.077655 (2.266401) | 2.079286 / 1.504120 (0.575166) | 1.887264 / 1.541195 (0.346069) | 1.910386 / 1.468490 (0.441896) | 0.538824 / 4.584777 (-4.045953) | 3.844786 / 3.745712 (0.099074) | 2.902091 / 5.269862 (-2.367770) | 1.270852 / 4.565676 (-3.294824) | 0.066324 / 0.424275 (-0.357951) | 0.011346 / 0.007607 (0.003739) | 0.537122 / 0.226044 (0.311078) | 5.367354 / 2.268929 (3.098426) | 2.533672 / 55.444624 (-52.910952) | 2.203260 / 6.876477 (-4.673217) | 2.224310 / 2.142072 (0.082237) | 0.663806 / 4.805227 (-4.141422) | 0.142758 / 6.500664 (-6.357906) | 0.063870 / 0.075469 (-0.011599) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.260487 / 1.841788 (-0.581301) | 14.800106 / 8.074308 (6.725798) | 13.993488 / 10.191392 (3.802096) | 0.165829 / 0.680424 (-0.514595) | 0.017347 / 0.534201 (-0.516854) | 0.401819 / 0.579283 (-0.177464) | 0.424577 / 0.434364 (-0.009787) | 0.475161 / 0.540337 (-0.065176) | 0.574659 / 1.386936 (-0.812277) |

![](https://cml.dev/watermark.png#02e1e9ab6df4720f57b2d08c0b800cecac79a7c8 "CML watermark")

Related to changes made in https://github.com/iterative/dvc/pull/9475