Improve logging

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007782 / 0.011353 (-0.003571) | 0.004451 / 0.011008 (-0.006557) | 0.099928 / 0.038508 (0.061420) | 0.081534 / 0.023109 (0.058425) | 0.379382 / 0.275898 (0.103484) | 0.410652 / 0.323480 (0.087172) | 0.005967 / 0.007986 (-0.002019) | 0.003702 / 0.004328 (-0.000627) | 0.076359 / 0.004250 (0.072109) | 0.066721 / 0.037052 (0.029669) | 0.383595 / 0.258489 (0.125106) | 0.423854 / 0.293841 (0.130013) | 0.032796 / 0.128546 (-0.095750) | 0.009728 / 0.075646 (-0.065918) | 0.344347 / 0.419271 (-0.074925) | 0.056320 / 0.043533 (0.012788) | 0.379974 / 0.255139 (0.124835) | 0.401294 / 0.283200 (0.118094) | 0.024110 / 0.141683 (-0.117572) | 1.804194 / 1.452155 (0.352039) | 1.860240 / 1.492716 (0.367523) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.233803 / 0.018006 (0.215797) | 0.506893 / 0.000490 (0.506404) | 0.003894 / 0.000200 (0.003694) | 0.000090 / 0.000054 (0.000035) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033328 / 0.037411 (-0.004083) | 0.098661 / 0.014526 (0.084136) | 0.114971 / 0.176557 (-0.061586) | 0.186815 / 0.737135 (-0.550321) | 0.115490 / 0.296338 (-0.180848) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.422590 / 0.215209 (0.207381) | 4.277189 / 2.077655 (2.199535) | 2.095565 / 1.504120 (0.591445) | 2.040825 / 1.541195 (0.499630) | 2.162562 / 1.468490 (0.694072) | 0.578602 / 4.584777 (-4.006175) | 4.203474 / 3.745712 (0.457762) | 6.674595 / 5.269862 (1.404734) | 3.913251 / 4.565676 (-0.652426) | 0.067777 / 0.424275 (-0.356498) | 0.008716 / 0.007607 (0.001109) | 0.548704 / 0.226044 (0.322660) | 5.162120 / 2.268929 (2.893192) | 2.600250 / 55.444624 (-52.844374) | 2.232730 / 6.876477 (-4.643747) | 2.485617 / 2.142072 (0.343544) | 0.650872 / 4.805227 (-4.154355) | 0.148022 / 6.500664 (-6.352642) | 0.064795 / 0.075469 (-0.010674) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.399439 / 1.841788 (-0.442349) | 22.438959 / 8.074308 (14.364651) | 16.447831 / 10.191392 (6.256439) | 0.202003 / 0.680424 (-0.478421) | 0.026200 / 0.534201 (-0.508001) | 0.472966 / 0.579283 (-0.106317) | 0.491621 / 0.434364 (0.057257) | 0.551580 / 0.540337 (0.011242) | 0.751420 / 1.386936 (-0.635516) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007241 / 0.011353 (-0.004112) | 0.004434 / 0.011008 (-0.006574) | 0.075872 / 0.038508 (0.037364) | 0.080094 / 0.023109 (0.056985) | 0.459244 / 0.275898 (0.183346) | 0.492482 / 0.323480 (0.169002) | 0.005791 / 0.007986 (-0.002194) | 0.003657 / 0.004328 (-0.000671) | 0.075214 / 0.004250 (0.070964) | 0.064208 / 0.037052 (0.027156) | 0.464195 / 0.258489 (0.205706) | 0.497809 / 0.293841 (0.203968) | 0.036301 / 0.128546 (-0.092245) | 0.009855 / 0.075646 (-0.065791) | 0.080826 / 0.419271 (-0.338445) | 0.056700 / 0.043533 (0.013167) | 0.452850 / 0.255139 (0.197711) | 0.490738 / 0.283200 (0.207538) | 0.024145 / 0.141683 (-0.117538) | 1.689911 / 1.452155 (0.237757) | 1.789803 / 1.492716 (0.297087) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.247741 / 0.018006 (0.229735) | 0.486769 / 0.000490 (0.486279) | 0.000418 / 0.000200 (0.000218) | 0.000060 / 0.000054 (0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.036317 / 0.037411 (-0.001094) | 0.104943 / 0.014526 (0.090417) | 0.120972 / 0.176557 (-0.055585) | 0.188461 / 0.737135 (-0.548674) | 0.120926 / 0.296338 (-0.175412) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.465788 / 0.215209 (0.250579) | 4.662369 / 2.077655 (2.584714) | 2.442241 / 1.504120 (0.938121) | 2.266328 / 1.541195 (0.725133) | 2.438998 / 1.468490 (0.970508) | 0.531384 / 4.584777 (-4.053393) | 4.125286 / 3.745712 (0.379574) | 3.920912 / 5.269862 (-1.348950) | 2.292149 / 4.565676 (-2.273528) | 0.070146 / 0.424275 (-0.354129) | 0.008887 / 0.007607 (0.001280) | 0.598181 / 0.226044 (0.372137) | 5.726454 / 2.268929 (3.457526) | 3.081836 / 55.444624 (-52.362788) | 2.683508 / 6.876477 (-4.192969) | 2.587350 / 2.142072 (0.445278) | 0.604736 / 4.805227 (-4.200491) | 0.141303 / 6.500664 (-6.359362) | 0.065020 / 0.075469 (-0.010449) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.481850 / 1.841788 (-0.359938) | 22.259592 / 8.074308 (14.185284) | 16.304290 / 10.191392 (6.112898) | 0.173514 / 0.680424 (-0.506909) | 0.021590 / 0.534201 (-0.512611) | 0.471753 / 0.579283 (-0.107531) | 0.472132 / 0.434364 (0.037768) | 0.563344 / 0.540337 (0.023007) | 0.738509 / 1.386936 (-0.648427) |

![](https://cml.dev/watermark.png#1cb7ae56dbd814945a4982c63bf0e50859a7b93a "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005910 / 0.011353 (-0.005443) | 0.004372 / 0.011008 (-0.006636) | 0.081583 / 0.038508 (0.043075) | 0.069598 / 0.023109 (0.046488) | 0.346360 / 0.275898 (0.070462) | 0.360733 / 0.323480 (0.037254) | 0.004725 / 0.007986 (-0.003261) | 0.003106 / 0.004328 (-0.001222) | 0.059916 / 0.004250 (0.055666) | 0.053242 / 0.037052 (0.016189) | 0.353551 / 0.258489 (0.095062) | 0.373052 / 0.293841 (0.079211) | 0.029036 / 0.128546 (-0.099510) | 0.007894 / 0.075646 (-0.067753) | 0.284131 / 0.419271 (-0.135140) | 0.049348 / 0.043533 (0.005815) | 0.347409 / 0.255139 (0.092270) | 0.355029 / 0.283200 (0.071830) | 0.022511 / 0.141683 (-0.119171) | 1.454495 / 1.452155 (0.002340) | 1.439551 / 1.492716 (-0.053166) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.218889 / 0.018006 (0.200883) | 0.478734 / 0.000490 (0.478244) | 0.003758 / 0.000200 (0.003558) | 0.000083 / 0.000054 (0.000029) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025759 / 0.037411 (-0.011653) | 0.082511 / 0.014526 (0.067985) | 0.087578 / 0.176557 (-0.088979) | 0.137760 / 0.737135 (-0.599375) | 0.093312 / 0.296338 (-0.203027) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.378963 / 0.215209 (0.163754) | 3.645846 / 2.077655 (1.568191) | 1.741135 / 1.504120 (0.237015) | 1.599166 / 1.541195 (0.057972) | 1.610817 / 1.468490 (0.142327) | 0.459209 / 4.584777 (-4.125568) | 3.484857 / 3.745712 (-0.260855) | 3.928109 / 5.269862 (-1.341752) | 2.419784 / 4.565676 (-2.145892) | 0.051987 / 0.424275 (-0.372288) | 0.006495 / 0.007607 (-0.001112) | 0.427311 / 0.226044 (0.201267) | 4.226378 / 2.268929 (1.957450) | 2.212331 / 55.444624 (-53.232293) | 1.916213 / 6.876477 (-4.960264) | 1.978809 / 2.142072 (-0.163263) | 0.547351 / 4.805227 (-4.257876) | 0.121110 / 6.500664 (-6.379554) | 0.054163 / 0.075469 (-0.021306) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.228594 / 1.841788 (-0.613193) | 19.410901 / 8.074308 (11.336593) | 13.014722 / 10.191392 (2.823330) | 0.156449 / 0.680424 (-0.523975) | 0.021032 / 0.534201 (-0.513169) | 0.403976 / 0.579283 (-0.175307) | 0.413885 / 0.434364 (-0.020479) | 0.470465 / 0.540337 (-0.069873) | 0.641322 / 1.386936 (-0.745614) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007210 / 0.011353 (-0.004143) | 0.003824 / 0.011008 (-0.007185) | 0.058227 / 0.038508 (0.019719) | 0.076211 / 0.023109 (0.053102) | 0.336626 / 0.275898 (0.060728) | 0.420542 / 0.323480 (0.097062) | 0.006178 / 0.007986 (-0.001808) | 0.003332 / 0.004328 (-0.000997) | 0.058073 / 0.004250 (0.053823) | 0.062485 / 0.037052 (0.025432) | 0.386175 / 0.258489 (0.127686) | 0.415659 / 0.293841 (0.121818) | 0.031264 / 0.128546 (-0.097282) | 0.007502 / 0.075646 (-0.068144) | 0.072079 / 0.419271 (-0.347192) | 0.055860 / 0.043533 (0.012327) | 0.343508 / 0.255139 (0.088369) | 0.437844 / 0.283200 (0.154645) | 0.032852 / 0.141683 (-0.108831) | 1.409241 / 1.452155 (-0.042913) | 1.623949 / 1.492716 (0.131233) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.207511 / 0.018006 (0.189504) | 0.464149 / 0.000490 (0.463660) | 0.003248 / 0.000200 (0.003048) | 0.000226 / 0.000054 (0.000172) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030767 / 0.037411 (-0.006645) | 0.079169 / 0.014526 (0.064643) | 0.093111 / 0.176557 (-0.083445) | 0.153369 / 0.737135 (-0.583767) | 0.092939 / 0.296338 (-0.203400) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.375602 / 0.215209 (0.160392) | 3.968612 / 2.077655 (1.890957) | 2.081749 / 1.504120 (0.577629) | 1.899772 / 1.541195 (0.358577) | 1.847923 / 1.468490 (0.379433) | 0.442867 / 4.584777 (-4.141910) | 3.646664 / 3.745712 (-0.099048) | 5.870600 / 5.269862 (0.600739) | 3.356698 / 4.565676 (-1.208979) | 0.051422 / 0.424275 (-0.372853) | 0.006006 / 0.007607 (-0.001601) | 0.442439 / 0.226044 (0.216395) | 4.466256 / 2.268929 (2.197328) | 2.483832 / 55.444624 (-52.960792) | 2.105612 / 6.876477 (-4.770865) | 2.060650 / 2.142072 (-0.081422) | 0.531119 / 4.805227 (-4.274108) | 0.123436 / 6.500664 (-6.377228) | 0.059838 / 0.075469 (-0.015632) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.283042 / 1.841788 (-0.558746) | 19.688251 / 8.074308 (11.613943) | 13.346386 / 10.191392 (3.154994) | 0.197463 / 0.680424 (-0.482961) | 0.018484 / 0.534201 (-0.515717) | 0.391727 / 0.579283 (-0.187556) | 0.425061 / 0.434364 (-0.009303) | 0.448177 / 0.540337 (-0.092160) | 0.653694 / 1.386936 (-0.733242) |

![](https://cml.dev/watermark.png#01604752fe89d290479fa406b1a24ac1f346826e "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008966 / 0.011353 (-0.002387) | 0.005195 / 0.011008 (-0.005813) | 0.102879 / 0.038508 (0.064371) | 0.090902 / 0.023109 (0.067792) | 0.434397 / 0.275898 (0.158498) | 0.454013 / 0.323480 (0.130534) | 0.008507 / 0.007986 (0.000521) | 0.005000 / 0.004328 (0.000671) | 0.075789 / 0.004250 (0.071538) | 0.067608 / 0.037052 (0.030555) | 0.435091 / 0.258489 (0.176602) | 0.469411 / 0.293841 (0.175570) | 0.050859 / 0.128546 (-0.077687) | 0.013560 / 0.075646 (-0.062086) | 0.345473 / 0.419271 (-0.073799) | 0.094974 / 0.043533 (0.051441) | 0.429626 / 0.255139 (0.174487) | 0.434290 / 0.283200 (0.151090) | 0.052269 / 0.141683 (-0.089413) | 1.700549 / 1.452155 (0.248395) | 1.890693 / 1.492716 (0.397976) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.296618 / 0.018006 (0.278612) | 0.613908 / 0.000490 (0.613419) | 0.000484 / 0.000200 (0.000284) | 0.000086 / 0.000054 (0.000032) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034346 / 0.037411 (-0.003065) | 0.096836 / 0.014526 (0.082310) | 0.113332 / 0.176557 (-0.063224) | 0.194464 / 0.737135 (-0.542671) | 0.111732 / 0.296338 (-0.184606) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.624954 / 0.215209 (0.409745) | 6.442193 / 2.077655 (4.364538) | 2.818331 / 1.504120 (1.314211) | 2.529607 / 1.541195 (0.988413) | 2.549026 / 1.468490 (1.080536) | 0.967367 / 4.584777 (-3.617410) | 5.446885 / 3.745712 (1.701173) | 6.259099 / 5.269862 (0.989237) | 3.652936 / 4.565676 (-0.912740) | 0.106420 / 0.424275 (-0.317855) | 0.011293 / 0.007607 (0.003686) | 0.772026 / 0.226044 (0.545982) | 7.823986 / 2.268929 (5.555057) | 3.725328 / 55.444624 (-51.719297) | 2.851489 / 6.876477 (-4.024988) | 3.013722 / 2.142072 (0.871649) | 1.045090 / 4.805227 (-3.760137) | 0.213174 / 6.500664 (-6.287490) | 0.077104 / 0.075469 (0.001635) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.657135 / 1.841788 (-0.184652) | 24.547604 / 8.074308 (16.473296) | 19.989533 / 10.191392 (9.798141) | 0.257139 / 0.680424 (-0.423285) | 0.028448 / 0.534201 (-0.505753) | 0.490801 / 0.579283 (-0.088482) | 0.628072 / 0.434364 (0.193708) | 0.584873 / 0.540337 (0.044536) | 0.825258 / 1.386936 (-0.561678) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009258 / 0.011353 (-0.002095) | 0.005660 / 0.011008 (-0.005348) | 0.080577 / 0.038508 (0.042069) | 0.095786 / 0.023109 (0.072676) | 0.473334 / 0.275898 (0.197436) | 0.527962 / 0.323480 (0.204482) | 0.006537 / 0.007986 (-0.001449) | 0.004411 / 0.004328 (0.000083) | 0.080702 / 0.004250 (0.076452) | 0.077020 / 0.037052 (0.039968) | 0.483205 / 0.258489 (0.224716) | 0.556916 / 0.293841 (0.263076) | 0.047670 / 0.128546 (-0.080877) | 0.016647 / 0.075646 (-0.058999) | 0.090653 / 0.419271 (-0.328619) | 0.062122 / 0.043533 (0.018589) | 0.498326 / 0.255139 (0.243187) | 0.546572 / 0.283200 (0.263372) | 0.037525 / 0.141683 (-0.104157) | 1.869520 / 1.452155 (0.417365) | 1.915335 / 1.492716 (0.422619) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.248287 / 0.018006 (0.230281) | 0.611440 / 0.000490 (0.610950) | 0.004102 / 0.000200 (0.003902) | 0.000132 / 0.000054 (0.000078) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.038228 / 0.037411 (0.000817) | 0.103510 / 0.014526 (0.088984) | 0.114337 / 0.176557 (-0.062219) | 0.189662 / 0.737135 (-0.547473) | 0.119078 / 0.296338 (-0.177260) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.606622 / 0.215209 (0.391413) | 6.053900 / 2.077655 (3.976246) | 2.857972 / 1.504120 (1.353852) | 2.549756 / 1.541195 (1.008561) | 2.584557 / 1.468490 (1.116067) | 0.930431 / 4.584777 (-3.654346) | 5.524077 / 3.745712 (1.778365) | 7.858406 / 5.269862 (2.588545) | 4.890697 / 4.565676 (0.325020) | 0.095356 / 0.424275 (-0.328919) | 0.008614 / 0.007607 (0.001007) | 0.774227 / 0.226044 (0.548182) | 7.470215 / 2.268929 (5.201287) | 3.784820 / 55.444624 (-51.659805) | 3.199364 / 6.876477 (-3.677113) | 3.212002 / 2.142072 (1.069929) | 1.054104 / 4.805227 (-3.751123) | 0.226044 / 6.500664 (-6.274620) | 0.092237 / 0.075469 (0.016768) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.801054 / 1.841788 (-0.040734) | 24.220404 / 8.074308 (16.146096) | 21.652936 / 10.191392 (11.461544) | 0.247004 / 0.680424 (-0.433420) | 0.029651 / 0.534201 (-0.504550) | 0.475702 / 0.579283 (-0.103581) | 0.621121 / 0.434364 (0.186757) | 0.570489 / 0.540337 (0.030151) | 0.768840 / 1.386936 (-0.618096) |

![](https://cml.dev/watermark.png#b2fc21eda345643fb57d1d1167ebed9043310911 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009223 / 0.011353 (-0.002130) | 0.005750 / 0.011008 (-0.005258) | 0.105264 / 0.038508 (0.066756) | 0.088478 / 0.023109 (0.065369) | 0.461119 / 0.275898 (0.185221) | 0.481115 / 0.323480 (0.157636) | 0.006366 / 0.007986 (-0.001619) | 0.004515 / 0.004328 (0.000186) | 0.079296 / 0.004250 (0.075045) | 0.063483 / 0.037052 (0.026430) | 0.444490 / 0.258489 (0.186001) | 0.496474 / 0.293841 (0.202634) | 0.048568 / 0.128546 (-0.079978) | 0.013574 / 0.075646 (-0.062073) | 0.379213 / 0.419271 (-0.040059) | 0.086464 / 0.043533 (0.042932) | 0.437526 / 0.255139 (0.182387) | 0.447117 / 0.283200 (0.163917) | 0.049502 / 0.141683 (-0.092180) | 1.749146 / 1.452155 (0.296992) | 1.831082 / 1.492716 (0.338365) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.268205 / 0.018006 (0.250199) | 0.627406 / 0.000490 (0.626917) | 0.005439 / 0.000200 (0.005239) | 0.000128 / 0.000054 (0.000074) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030564 / 0.037411 (-0.006848) | 0.096365 / 0.014526 (0.081840) | 0.117484 / 0.176557 (-0.059072) | 0.189104 / 0.737135 (-0.548032) | 0.118073 / 0.296338 (-0.178266) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.618229 / 0.215209 (0.403019) | 6.437853 / 2.077655 (4.360199) | 2.789946 / 1.504120 (1.285826) | 2.339245 / 1.541195 (0.798050) | 2.588779 / 1.468490 (1.120289) | 0.921008 / 4.584777 (-3.663769) | 5.402940 / 3.745712 (1.657227) | 4.818783 / 5.269862 (-0.451078) | 3.162259 / 4.565676 (-1.403417) | 0.108501 / 0.424275 (-0.315774) | 0.009384 / 0.007607 (0.001777) | 0.766811 / 0.226044 (0.540766) | 7.624629 / 2.268929 (5.355701) | 3.442420 / 55.444624 (-52.002204) | 2.759967 / 6.876477 (-4.116510) | 3.049644 / 2.142072 (0.907572) | 1.113308 / 4.805227 (-3.691919) | 0.223923 / 6.500664 (-6.276741) | 0.079156 / 0.075469 (0.003687) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.683318 / 1.841788 (-0.158470) | 25.062141 / 8.074308 (16.987833) | 21.777131 / 10.191392 (11.585739) | 0.266939 / 0.680424 (-0.413485) | 0.029670 / 0.534201 (-0.504531) | 0.476761 / 0.579283 (-0.102522) | 0.622080 / 0.434364 (0.187716) | 0.601781 / 0.540337 (0.061443) | 0.785126 / 1.386936 (-0.601811) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.010198 / 0.011353 (-0.001155) | 0.005777 / 0.011008 (-0.005231) | 0.083003 / 0.038508 (0.044495) | 0.093411 / 0.023109 (0.070302) | 0.496178 / 0.275898 (0.220280) | 0.554670 / 0.323480 (0.231190) | 0.008351 / 0.007986 (0.000365) | 0.004678 / 0.004328 (0.000350) | 0.083631 / 0.004250 (0.079381) | 0.075538 / 0.037052 (0.038485) | 0.492410 / 0.258489 (0.233921) | 0.545209 / 0.293841 (0.251368) | 0.048365 / 0.128546 (-0.080181) | 0.014219 / 0.075646 (-0.061427) | 0.100749 / 0.419271 (-0.318523) | 0.063431 / 0.043533 (0.019898) | 0.511115 / 0.255139 (0.255976) | 0.532965 / 0.283200 (0.249765) | 0.037968 / 0.141683 (-0.103715) | 1.940268 / 1.452155 (0.488113) | 2.032934 / 1.492716 (0.540217) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.238179 / 0.018006 (0.220172) | 0.605767 / 0.000490 (0.605277) | 0.004033 / 0.000200 (0.003833) | 0.000125 / 0.000054 (0.000071) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.036436 / 0.037411 (-0.000975) | 0.108034 / 0.014526 (0.093509) | 0.118624 / 0.176557 (-0.057933) | 0.183079 / 0.737135 (-0.554056) | 0.121739 / 0.296338 (-0.174600) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.630538 / 0.215209 (0.415329) | 6.552184 / 2.077655 (4.474529) | 3.003412 / 1.504120 (1.499292) | 2.669026 / 1.541195 (1.127832) | 2.791109 / 1.468490 (1.322619) | 0.884003 / 4.584777 (-3.700774) | 5.538660 / 3.745712 (1.792947) | 5.126708 / 5.269862 (-0.143154) | 3.120825 / 4.565676 (-1.444852) | 0.101178 / 0.424275 (-0.323097) | 0.009027 / 0.007607 (0.001420) | 0.785914 / 0.226044 (0.559869) | 7.994720 / 2.268929 (5.725792) | 4.061996 / 55.444624 (-51.382629) | 3.263230 / 6.876477 (-3.613247) | 3.288622 / 2.142072 (1.146550) | 1.141867 / 4.805227 (-3.663360) | 0.255287 / 6.500664 (-6.245378) | 0.100637 / 0.075469 (0.025168) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.769821 / 1.841788 (-0.071967) | 24.994008 / 8.074308 (16.919700) | 21.765971 / 10.191392 (11.574579) | 0.268493 / 0.680424 (-0.411931) | 0.028047 / 0.534201 (-0.506154) | 0.489472 / 0.579283 (-0.089811) | 0.594809 / 0.434364 (0.160445) | 0.613578 / 0.540337 (0.073241) | 0.879360 / 1.386936 (-0.507576) |

![](https://cml.dev/watermark.png#b85b1154aef2a9ab4d558f60d91623f2cc1583c4 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006003 / 0.011353 (-0.005350) | 0.003590 / 0.011008 (-0.007418) | 0.084657 / 0.038508 (0.046149) | 0.057884 / 0.023109 (0.034775) | 0.318347 / 0.275898 (0.042449) | 0.345976 / 0.323480 (0.022496) | 0.004706 / 0.007986 (-0.003279) | 0.002921 / 0.004328 (-0.001407) | 0.061850 / 0.004250 (0.057600) | 0.050558 / 0.037052 (0.013505) | 0.320877 / 0.258489 (0.062388) | 0.356062 / 0.293841 (0.062222) | 0.027511 / 0.128546 (-0.101035) | 0.007954 / 0.075646 (-0.067693) | 0.260290 / 0.419271 (-0.158981) | 0.051207 / 0.043533 (0.007674) | 0.334423 / 0.255139 (0.079284) | 0.338575 / 0.283200 (0.055375) | 0.022330 / 0.141683 (-0.119353) | 1.445446 / 1.452155 (-0.006709) | 1.500626 / 1.492716 (0.007910) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.192440 / 0.018006 (0.174433) | 0.428455 / 0.000490 (0.427965) | 0.000318 / 0.000200 (0.000118) | 0.000056 / 0.000054 (0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022933 / 0.037411 (-0.014478) | 0.072795 / 0.014526 (0.058269) | 0.081149 / 0.176557 (-0.095407) | 0.142941 / 0.737135 (-0.594195) | 0.082410 / 0.296338 (-0.213928) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.405220 / 0.215209 (0.190011) | 4.048585 / 2.077655 (1.970931) | 2.027908 / 1.504120 (0.523788) | 1.887828 / 1.541195 (0.346633) | 2.131780 / 1.468490 (0.663290) | 0.502847 / 4.584777 (-4.081930) | 3.069498 / 3.745712 (-0.676215) | 4.094774 / 5.269862 (-1.175088) | 2.544004 / 4.565676 (-2.021673) | 0.059540 / 0.424275 (-0.364735) | 0.006501 / 0.007607 (-0.001106) | 0.477218 / 0.226044 (0.251173) | 4.764961 / 2.268929 (2.496032) | 2.434594 / 55.444624 (-53.010030) | 2.104833 / 6.876477 (-4.771644) | 2.263059 / 2.142072 (0.120987) | 0.591755 / 4.805227 (-4.213472) | 0.131167 / 6.500664 (-6.369497) | 0.061808 / 0.075469 (-0.013661) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.345364 / 1.841788 (-0.496424) | 18.122584 / 8.074308 (10.048276) | 13.318689 / 10.191392 (3.127297) | 0.144526 / 0.680424 (-0.535898) | 0.016997 / 0.534201 (-0.517204) | 0.336036 / 0.579283 (-0.243247) | 0.359532 / 0.434364 (-0.074832) | 0.386945 / 0.540337 (-0.153392) | 0.538659 / 1.386936 (-0.848277) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006088 / 0.011353 (-0.005265) | 0.003684 / 0.011008 (-0.007324) | 0.062340 / 0.038508 (0.023832) | 0.058461 / 0.023109 (0.035352) | 0.360134 / 0.275898 (0.084236) | 0.393298 / 0.323480 (0.069818) | 0.004664 / 0.007986 (-0.003322) | 0.002909 / 0.004328 (-0.001420) | 0.062668 / 0.004250 (0.058418) | 0.050145 / 0.037052 (0.013092) | 0.361897 / 0.258489 (0.103408) | 0.402008 / 0.293841 (0.108167) | 0.027491 / 0.128546 (-0.101055) | 0.008113 / 0.075646 (-0.067534) | 0.068114 / 0.419271 (-0.351157) | 0.043303 / 0.043533 (-0.000230) | 0.360569 / 0.255139 (0.105430) | 0.387144 / 0.283200 (0.103944) | 0.020194 / 0.141683 (-0.121489) | 1.418066 / 1.452155 (-0.034089) | 1.475640 / 1.492716 (-0.017076) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.200291 / 0.018006 (0.182285) | 0.432298 / 0.000490 (0.431809) | 0.003303 / 0.000200 (0.003103) | 0.000075 / 0.000054 (0.000020) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027749 / 0.037411 (-0.009662) | 0.081890 / 0.014526 (0.067364) | 0.094319 / 0.176557 (-0.082238) | 0.148646 / 0.737135 (-0.588490) | 0.091830 / 0.296338 (-0.204509) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.433546 / 0.215209 (0.218337) | 4.326855 / 2.077655 (2.249200) | 2.230186 / 1.504120 (0.726066) | 2.052524 / 1.541195 (0.511329) | 2.117270 / 1.468490 (0.648779) | 0.500331 / 4.584777 (-4.084446) | 3.113662 / 3.745712 (-0.632050) | 2.931540 / 5.269862 (-2.338322) | 1.853615 / 4.565676 (-2.712062) | 0.058250 / 0.424275 (-0.366025) | 0.006546 / 0.007607 (-0.001061) | 0.508850 / 0.226044 (0.282806) | 5.081809 / 2.268929 (2.812880) | 2.687037 / 55.444624 (-52.757588) | 2.369317 / 6.876477 (-4.507160) | 2.383549 / 2.142072 (0.241477) | 0.587039 / 4.805227 (-4.218188) | 0.125858 / 6.500664 (-6.374806) | 0.062522 / 0.075469 (-0.012947) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.294929 / 1.841788 (-0.546858) | 18.056312 / 8.074308 (9.982004) | 13.755117 / 10.191392 (3.563725) | 0.132037 / 0.680424 (-0.548387) | 0.016866 / 0.534201 (-0.517335) | 0.339040 / 0.579283 (-0.240243) | 0.364371 / 0.434364 (-0.069993) | 0.399533 / 0.540337 (-0.140804) | 0.564524 / 1.386936 (-0.822412) |

![](https://cml.dev/watermark.png#64b811c13a7982015d7e078e3d693ce5359a05a2 "CML watermark")

@lhoestq This bar comes from: https://github.com/huggingface/datasets/blob/b8067c0262073891180869f700ebef5ac3dc5cce/src/datasets/builder.py#L1156-L1166

Do you prefer not showing it or, e.g.,  having `desc="Generating splits"`?

No strong opinion. Since there is a "Generating" progress bar already, maybe it can be "Preparing splits" (ref to download_and_prepare)

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006348 / 0.011353 (-0.005005) | 0.003721 / 0.011008 (-0.007287) | 0.084039 / 0.038508 (0.045531) | 0.067627 / 0.023109 (0.044517) | 0.308372 / 0.275898 (0.032474) | 0.335131 / 0.323480 (0.011652) | 0.005157 / 0.007986 (-0.002829) | 0.003266 / 0.004328 (-0.001062) | 0.065374 / 0.004250 (0.061124) | 0.055550 / 0.037052 (0.018498) | 0.314001 / 0.258489 (0.055512) | 0.350510 / 0.293841 (0.056669) | 0.030859 / 0.128546 (-0.097688) | 0.008286 / 0.075646 (-0.067361) | 0.287122 / 0.419271 (-0.132149) | 0.051494 / 0.043533 (0.007961) | 0.309868 / 0.255139 (0.054729) | 0.325845 / 0.283200 (0.042645) | 0.022622 / 0.141683 (-0.119061) | 1.468730 / 1.452155 (0.016575) | 1.547871 / 1.492716 (0.055155) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.202763 / 0.018006 (0.184757) | 0.456403 / 0.000490 (0.455914) | 0.003116 / 0.000200 (0.002916) | 0.000079 / 0.000054 (0.000024) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027297 / 0.037411 (-0.010114) | 0.081204 / 0.014526 (0.066678) | 0.094274 / 0.176557 (-0.082282) | 0.154391 / 0.737135 (-0.582744) | 0.094312 / 0.296338 (-0.202026) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.387382 / 0.215209 (0.172173) | 3.865597 / 2.077655 (1.787943) | 1.855959 / 1.504120 (0.351839) | 1.685411 / 1.541195 (0.144216) | 1.732127 / 1.468490 (0.263637) | 0.482230 / 4.584777 (-4.102547) | 3.664947 / 3.745712 (-0.080765) | 5.114379 / 5.269862 (-0.155482) | 3.102803 / 4.565676 (-1.462873) | 0.056509 / 0.424275 (-0.367766) | 0.007230 / 0.007607 (-0.000377) | 0.456788 / 0.226044 (0.230744) | 4.575831 / 2.268929 (2.306902) | 2.335249 / 55.444624 (-53.109375) | 2.003805 / 6.876477 (-4.872672) | 2.141788 / 2.142072 (-0.000285) | 0.577501 / 4.805227 (-4.227726) | 0.130264 / 6.500664 (-6.370400) | 0.058889 / 0.075469 (-0.016580) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.252673 / 1.841788 (-0.589115) | 18.676897 / 8.074308 (10.602589) | 13.988101 / 10.191392 (3.796709) | 0.151376 / 0.680424 (-0.529048) | 0.018104 / 0.534201 (-0.516097) | 0.388413 / 0.579283 (-0.190870) | 0.414841 / 0.434364 (-0.019523) | 0.456078 / 0.540337 (-0.084259) | 0.641715 / 1.386936 (-0.745221) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006315 / 0.011353 (-0.005038) | 0.003847 / 0.011008 (-0.007162) | 0.063989 / 0.038508 (0.025481) | 0.068244 / 0.023109 (0.045135) | 0.416201 / 0.275898 (0.140303) | 0.438446 / 0.323480 (0.114966) | 0.005820 / 0.007986 (-0.002166) | 0.003165 / 0.004328 (-0.001163) | 0.064143 / 0.004250 (0.059892) | 0.056529 / 0.037052 (0.019477) | 0.414916 / 0.258489 (0.156427) | 0.450771 / 0.293841 (0.156930) | 0.030611 / 0.128546 (-0.097935) | 0.008289 / 0.075646 (-0.067357) | 0.070725 / 0.419271 (-0.348546) | 0.047998 / 0.043533 (0.004465) | 0.405609 / 0.255139 (0.150470) | 0.421895 / 0.283200 (0.138696) | 0.022135 / 0.141683 (-0.119548) | 1.444238 / 1.452155 (-0.007916) | 1.515823 / 1.492716 (0.023107) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.227043 / 0.018006 (0.209037) | 0.439732 / 0.000490 (0.439242) | 0.001267 / 0.000200 (0.001067) | 0.000070 / 0.000054 (0.000016) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029082 / 0.037411 (-0.008329) | 0.086201 / 0.014526 (0.071675) | 0.098653 / 0.176557 (-0.077903) | 0.152574 / 0.737135 (-0.584561) | 0.100696 / 0.296338 (-0.195642) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.411243 / 0.215209 (0.196034) | 4.100170 / 2.077655 (2.022515) | 2.118310 / 1.504120 (0.614190) | 1.935646 / 1.541195 (0.394451) | 1.970798 / 1.468490 (0.502307) | 0.478635 / 4.584777 (-4.106142) | 3.589396 / 3.745712 (-0.156316) | 3.312462 / 5.269862 (-1.957399) | 1.963081 / 4.565676 (-2.602595) | 0.056392 / 0.424275 (-0.367883) | 0.007134 / 0.007607 (-0.000473) | 0.485131 / 0.226044 (0.259086) | 4.838946 / 2.268929 (2.570017) | 2.624550 / 55.444624 (-52.820075) | 2.223046 / 6.876477 (-4.653431) | 2.230642 / 2.142072 (0.088570) | 0.594892 / 4.805227 (-4.210335) | 0.130523 / 6.500664 (-6.370141) | 0.059585 / 0.075469 (-0.015884) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.329941 / 1.841788 (-0.511847) | 19.199057 / 8.074308 (11.124748) | 14.166009 / 10.191392 (3.974617) | 0.190595 / 0.680424 (-0.489829) | 0.018419 / 0.534201 (-0.515782) | 0.392031 / 0.579283 (-0.187252) | 0.409395 / 0.434364 (-0.024969) | 0.475930 / 0.540337 (-0.064408) | 0.654412 / 1.386936 (-0.732524) |

![](https://cml.dev/watermark.png#42fdfbd567674d075c3a9148ec3c95221eb62cfe "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007500 / 0.011353 (-0.003853) | 0.004328 / 0.011008 (-0.006681) | 0.086718 / 0.038508 (0.048209) | 0.098638 / 0.023109 (0.075529) | 0.335308 / 0.275898 (0.059409) | 0.369163 / 0.323480 (0.045683) | 0.005733 / 0.007986 (-0.002253) | 0.003738 / 0.004328 (-0.000590) | 0.066452 / 0.004250 (0.062202) | 0.066245 / 0.037052 (0.029192) | 0.337609 / 0.258489 (0.079120) | 0.388584 / 0.293841 (0.094744) | 0.031742 / 0.128546 (-0.096804) | 0.008721 / 0.075646 (-0.066925) | 0.290820 / 0.419271 (-0.128452) | 0.053323 / 0.043533 (0.009790) | 0.329192 / 0.255139 (0.074053) | 0.350560 / 0.283200 (0.067360) | 0.025402 / 0.141683 (-0.116281) | 1.476174 / 1.452155 (0.024020) | 1.578194 / 1.492716 (0.085478) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.256160 / 0.018006 (0.238154) | 0.560315 / 0.000490 (0.559825) | 0.005287 / 0.000200 (0.005088) | 0.000094 / 0.000054 (0.000040) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029164 / 0.037411 (-0.008247) | 0.084881 / 0.014526 (0.070356) | 0.100979 / 0.176557 (-0.075577) | 0.156539 / 0.737135 (-0.580597) | 0.101510 / 0.296338 (-0.194828) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.381138 / 0.215209 (0.165929) | 3.791573 / 2.077655 (1.713918) | 1.841954 / 1.504120 (0.337834) | 1.672463 / 1.541195 (0.131268) | 1.785769 / 1.468490 (0.317279) | 0.483263 / 4.584777 (-4.101514) | 3.617391 / 3.745712 (-0.128322) | 5.607794 / 5.269862 (0.337933) | 3.359530 / 4.565676 (-1.206147) | 0.056826 / 0.424275 (-0.367449) | 0.007375 / 0.007607 (-0.000232) | 0.455853 / 0.226044 (0.229809) | 4.548965 / 2.268929 (2.280037) | 2.412716 / 55.444624 (-53.031908) | 1.991456 / 6.876477 (-4.885021) | 2.242851 / 2.142072 (0.100778) | 0.573070 / 4.805227 (-4.232157) | 0.134658 / 6.500664 (-6.366006) | 0.061539 / 0.075469 (-0.013930) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.278306 / 1.841788 (-0.563481) | 20.634317 / 8.074308 (12.560009) | 15.164246 / 10.191392 (4.972854) | 0.167487 / 0.680424 (-0.512937) | 0.019006 / 0.534201 (-0.515195) | 0.394617 / 0.579283 (-0.184666) | 0.423385 / 0.434364 (-0.010979) | 0.469968 / 0.540337 (-0.070370) | 0.630058 / 1.386936 (-0.756878) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006793 / 0.011353 (-0.004559) | 0.004260 / 0.011008 (-0.006748) | 0.065398 / 0.038508 (0.026890) | 0.077850 / 0.023109 (0.054741) | 0.371754 / 0.275898 (0.095855) | 0.400652 / 0.323480 (0.077172) | 0.005729 / 0.007986 (-0.002256) | 0.003660 / 0.004328 (-0.000669) | 0.065119 / 0.004250 (0.060869) | 0.060714 / 0.037052 (0.023661) | 0.384592 / 0.258489 (0.126103) | 0.412806 / 0.293841 (0.118965) | 0.031865 / 0.128546 (-0.096681) | 0.008807 / 0.075646 (-0.066839) | 0.071156 / 0.419271 (-0.348115) | 0.049571 / 0.043533 (0.006038) | 0.367381 / 0.255139 (0.112242) | 0.386713 / 0.283200 (0.103513) | 0.024838 / 0.141683 (-0.116845) | 1.492986 / 1.452155 (0.040831) | 1.559243 / 1.492716 (0.066526) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.269737 / 0.018006 (0.251730) | 0.565177 / 0.000490 (0.564687) | 0.000404 / 0.000200 (0.000204) | 0.000060 / 0.000054 (0.000006) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031631 / 0.037411 (-0.005780) | 0.087289 / 0.014526 (0.072764) | 0.102798 / 0.176557 (-0.073759) | 0.158977 / 0.737135 (-0.578158) | 0.105495 / 0.296338 (-0.190843) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.425067 / 0.215209 (0.209858) | 4.243121 / 2.077655 (2.165466) | 2.234567 / 1.504120 (0.730447) | 2.070810 / 1.541195 (0.529615) | 2.176802 / 1.468490 (0.708312) | 0.484987 / 4.584777 (-4.099790) | 3.647000 / 3.745712 (-0.098712) | 3.574843 / 5.269862 (-1.695019) | 2.092581 / 4.565676 (-2.473095) | 0.057299 / 0.424275 (-0.366976) | 0.007480 / 0.007607 (-0.000128) | 0.507838 / 0.226044 (0.281794) | 5.076594 / 2.268929 (2.807666) | 2.718858 / 55.444624 (-52.725766) | 2.362793 / 6.876477 (-4.513684) | 2.451962 / 2.142072 (0.309890) | 0.581355 / 4.805227 (-4.223872) | 0.133723 / 6.500664 (-6.366941) | 0.061896 / 0.075469 (-0.013573) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.325814 / 1.841788 (-0.515974) | 20.614502 / 8.074308 (12.540194) | 14.769422 / 10.191392 (4.578029) | 0.193797 / 0.680424 (-0.486627) | 0.018379 / 0.534201 (-0.515822) | 0.394153 / 0.579283 (-0.185130) | 0.409585 / 0.434364 (-0.024779) | 0.479107 / 0.540337 (-0.061231) | 0.668397 / 1.386936 (-0.718539) |

![](https://cml.dev/watermark.png#b2d892237169bad5512c91cae453d257ebefc201 "CML watermark")

In the end, I decided to remove the progress bar to avoid having it displayed when loading a cached dataset.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006673 / 0.011353 (-0.004680) | 0.004162 / 0.011008 (-0.006846) | 0.084017 / 0.038508 (0.045509) | 0.079536 / 0.023109 (0.056426) | 0.313594 / 0.275898 (0.037695) | 0.349200 / 0.323480 (0.025720) | 0.005544 / 0.007986 (-0.002441) | 0.003472 / 0.004328 (-0.000857) | 0.064742 / 0.004250 (0.060491) | 0.056857 / 0.037052 (0.019805) | 0.318635 / 0.258489 (0.060146) | 0.354378 / 0.293841 (0.060537) | 0.030856 / 0.128546 (-0.097690) | 0.008759 / 0.075646 (-0.066887) | 0.287760 / 0.419271 (-0.131511) | 0.052307 / 0.043533 (0.008775) | 0.316396 / 0.255139 (0.061257) | 0.351408 / 0.283200 (0.068208) | 0.024914 / 0.141683 (-0.116769) | 1.484592 / 1.452155 (0.032437) | 1.560662 / 1.492716 (0.067945) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.280938 / 0.018006 (0.262932) | 0.580236 / 0.000490 (0.579747) | 0.003369 / 0.000200 (0.003169) | 0.000090 / 0.000054 (0.000036) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028736 / 0.037411 (-0.008675) | 0.082916 / 0.014526 (0.068390) | 0.097761 / 0.176557 (-0.078796) | 0.153515 / 0.737135 (-0.583620) | 0.099282 / 0.296338 (-0.197057) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.401244 / 0.215209 (0.186035) | 4.019866 / 2.077655 (1.942211) | 2.029642 / 1.504120 (0.525522) | 1.849591 / 1.541195 (0.308396) | 1.946829 / 1.468490 (0.478339) | 0.479750 / 4.584777 (-4.105027) | 3.482822 / 3.745712 (-0.262890) | 3.955859 / 5.269862 (-1.314003) | 2.370747 / 4.565676 (-2.194930) | 0.056905 / 0.424275 (-0.367370) | 0.007319 / 0.007607 (-0.000288) | 0.485310 / 0.226044 (0.259266) | 4.858228 / 2.268929 (2.589299) | 2.500476 / 55.444624 (-52.944148) | 2.171156 / 6.876477 (-4.705320) | 2.427266 / 2.142072 (0.285194) | 0.570199 / 4.805227 (-4.235029) | 0.130855 / 6.500664 (-6.369809) | 0.060269 / 0.075469 (-0.015200) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.258044 / 1.841788 (-0.583743) | 20.218657 / 8.074308 (12.144349) | 13.597970 / 10.191392 (3.406578) | 0.167656 / 0.680424 (-0.512768) | 0.018137 / 0.534201 (-0.516064) | 0.395309 / 0.579283 (-0.183975) | 0.406325 / 0.434364 (-0.028039) | 0.467457 / 0.540337 (-0.072880) | 0.613636 / 1.386936 (-0.773300) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006846 / 0.011353 (-0.004507) | 0.004207 / 0.011008 (-0.006802) | 0.064525 / 0.038508 (0.026017) | 0.081329 / 0.023109 (0.058220) | 0.399838 / 0.275898 (0.123940) | 0.431305 / 0.323480 (0.107825) | 0.005859 / 0.007986 (-0.002127) | 0.003568 / 0.004328 (-0.000760) | 0.065262 / 0.004250 (0.061011) | 0.064796 / 0.037052 (0.027744) | 0.406858 / 0.258489 (0.148369) | 0.440971 / 0.293841 (0.147130) | 0.031421 / 0.128546 (-0.097125) | 0.008777 / 0.075646 (-0.066870) | 0.071418 / 0.419271 (-0.347853) | 0.049263 / 0.043533 (0.005730) | 0.384279 / 0.255139 (0.129140) | 0.410745 / 0.283200 (0.127546) | 0.024467 / 0.141683 (-0.117216) | 1.522379 / 1.452155 (0.070224) | 1.581636 / 1.492716 (0.088920) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.276161 / 0.018006 (0.258155) | 0.548842 / 0.000490 (0.548352) | 0.004523 / 0.000200 (0.004324) | 0.000098 / 0.000054 (0.000043) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030747 / 0.037411 (-0.006664) | 0.087493 / 0.014526 (0.072967) | 0.106563 / 0.176557 (-0.069993) | 0.162949 / 0.737135 (-0.574186) | 0.105303 / 0.296338 (-0.191036) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.425854 / 0.215209 (0.210645) | 4.244797 / 2.077655 (2.167142) | 2.269006 / 1.504120 (0.764886) | 2.097428 / 1.541195 (0.556234) | 2.181038 / 1.468490 (0.712548) | 0.477286 / 4.584777 (-4.107491) | 3.591452 / 3.745712 (-0.154260) | 3.481281 / 5.269862 (-1.788580) | 2.066895 / 4.565676 (-2.498782) | 0.056576 / 0.424275 (-0.367699) | 0.007409 / 0.007607 (-0.000199) | 0.498411 / 0.226044 (0.272367) | 4.994873 / 2.268929 (2.725945) | 2.749148 / 55.444624 (-52.695476) | 2.378544 / 6.876477 (-4.497932) | 2.452859 / 2.142072 (0.310786) | 0.571340 / 4.805227 (-4.233887) | 0.132174 / 6.500664 (-6.368490) | 0.061507 / 0.075469 (-0.013962) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.370773 / 1.841788 (-0.471015) | 20.493342 / 8.074308 (12.419034) | 14.809886 / 10.191392 (4.618494) | 0.175730 / 0.680424 (-0.504693) | 0.018617 / 0.534201 (-0.515583) | 0.393808 / 0.579283 (-0.185476) | 0.416419 / 0.434364 (-0.017945) | 0.477183 / 0.540337 (-0.063155) | 0.668060 / 1.386936 (-0.718876) |

![](https://cml.dev/watermark.png#2de7a2a4af5d94b0f98a7a6db94e78984af40602 "CML watermark")

Adds the StreamHandler (as `hfh` and `transformers` do) to the library's logger to log INFO messages and logs the messages about "loading a cached result" (and some other warnings) as INFO

(Also removes the `leave=False` arg in the progress bars to be consistent with `hfh` and `transformers` - progress bars serve as an indicator that a result is not cached, so it makes more sense not to delete them)

Fix #2832, fix https://github.com/huggingface/datasets/issues/1948, fix https://github.com/huggingface/datasets/issues/5444


test1

We no longer host datasets in this repo. You should use the HF Hub instead.


Switch to huggingface_hub's HfFileSystem

enhancement

New feature or request

instead of the current datasets.filesystems.hffilesystem.HfFileSystem which can be slow in some cases

related to https://github.com/huggingface/datasets/issues/5846 and https://github.com/huggingface/datasets/pull/5919


Dataset string representation enhancement

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6016). All of your documentation changes will be reflected on that endpoint.

It we could have something similar to Polars, that would be great.

This is what Polars outputs:
* `__repr__`/`__str__` :
```
shape: (67_349, 3)
┌───────┬───────────────────────────────────┬───────┐
│ idx   ┆ sentence                          ┆ label │
│ ---   ┆ ---                               ┆ ---   │
│ i32   ┆ str                               ┆ i64   │
╞═══════╪═══════════════════════════════════╪═══════╡
│ 0     ┆ hide new secretions from the par… ┆ 0     │
│ 1     ┆ contains no wit , only labored g… ┆ 0     │
│ 2     ┆ that loves its characters and co… ┆ 1     │
│ 3     ┆ remains utterly satisfied to rem… ┆ 0     │
│ …     ┆ …                                 ┆ …     │
│ 67345 ┆ anguish , anger and frustration   ┆ 0     │
│ 67346 ┆ at achieving the modest , crowd-… ┆ 1     │
│ 67347 ┆ a patient viewer                  ┆ 1     │
│ 67348 ┆ this new jangle of noise , mayhe… ┆ 0     │
└───────┴───────────────────────────────────┴───────┘
```

* `_repr_html_`:
<img width="251" alt="Screenshot 2023-07-12 at 18 25 58" src="https://github.com/huggingface/datasets/assets/47462742/5d04519d-f302-4411-9fbc-7445bdf53b23">

my attempt at #6010
not sure if this is the right way to go about it, I will wait for your feedback


Add metadata ui screenshot in docs

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007633 / 0.011353 (-0.003720) | 0.004666 / 0.011008 (-0.006343) | 0.097768 / 0.038508 (0.059260) | 0.085153 / 0.023109 (0.062044) | 0.400315 / 0.275898 (0.124417) | 0.452903 / 0.323480 (0.129423) | 0.006227 / 0.007986 (-0.001759) | 0.003814 / 0.004328 (-0.000515) | 0.074586 / 0.004250 (0.070336) | 0.064295 / 0.037052 (0.027242) | 0.408082 / 0.258489 (0.149593) | 0.446921 / 0.293841 (0.153080) | 0.034593 / 0.128546 (-0.093953) | 0.009191 / 0.075646 (-0.066456) | 0.337099 / 0.419271 (-0.082173) | 0.075320 / 0.043533 (0.031787) | 0.403488 / 0.255139 (0.148349) | 0.435309 / 0.283200 (0.152109) | 0.035675 / 0.141683 (-0.106008) | 1.732642 / 1.452155 (0.280487) | 1.770238 / 1.492716 (0.277522) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.235879 / 0.018006 (0.217873) | 0.500330 / 0.000490 (0.499841) | 0.005221 / 0.000200 (0.005021) | 0.000150 / 0.000054 (0.000096) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032479 / 0.037411 (-0.004933) | 0.095873 / 0.014526 (0.081348) | 0.107118 / 0.176557 (-0.069438) | 0.173809 / 0.737135 (-0.563326) | 0.109832 / 0.296338 (-0.186507) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.444342 / 0.215209 (0.229133) | 4.459010 / 2.077655 (2.381355) | 2.209687 / 1.504120 (0.705567) | 2.007556 / 1.541195 (0.466362) | 2.113683 / 1.468490 (0.645193) | 0.544281 / 4.584777 (-4.040496) | 4.037151 / 3.745712 (0.291439) | 4.852644 / 5.269862 (-0.417217) | 3.134126 / 4.565676 (-1.431550) | 0.066815 / 0.424275 (-0.357460) | 0.008836 / 0.007607 (0.001229) | 0.560904 / 0.226044 (0.334859) | 5.302760 / 2.268929 (3.033832) | 2.750182 / 55.444624 (-52.694442) | 2.322595 / 6.876477 (-4.553882) | 2.547486 / 2.142072 (0.405414) | 0.665766 / 4.805227 (-4.139461) | 0.151613 / 6.500664 (-6.349051) | 0.071155 / 0.075469 (-0.004314) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.473717 / 1.841788 (-0.368071) | 22.584179 / 8.074308 (14.509871) | 15.888001 / 10.191392 (5.696609) | 0.181073 / 0.680424 (-0.499351) | 0.021395 / 0.534201 (-0.512806) | 0.452693 / 0.579283 (-0.126590) | 0.447709 / 0.434364 (0.013345) | 0.529599 / 0.540337 (-0.010738) | 0.699241 / 1.386936 (-0.687695) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007917 / 0.011353 (-0.003436) | 0.004544 / 0.011008 (-0.006464) | 0.074566 / 0.038508 (0.036058) | 0.087530 / 0.023109 (0.064421) | 0.419753 / 0.275898 (0.143854) | 0.452352 / 0.323480 (0.128872) | 0.005882 / 0.007986 (-0.002104) | 0.003904 / 0.004328 (-0.000425) | 0.073539 / 0.004250 (0.069289) | 0.071320 / 0.037052 (0.034267) | 0.432899 / 0.258489 (0.174409) | 0.470365 / 0.293841 (0.176524) | 0.036198 / 0.128546 (-0.092348) | 0.009342 / 0.075646 (-0.066304) | 0.080970 / 0.419271 (-0.338301) | 0.058769 / 0.043533 (0.015236) | 0.413397 / 0.255139 (0.158258) | 0.448362 / 0.283200 (0.165162) | 0.034177 / 0.141683 (-0.107506) | 1.706217 / 1.452155 (0.254063) | 1.776743 / 1.492716 (0.284026) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.198779 / 0.018006 (0.180773) | 0.499862 / 0.000490 (0.499372) | 0.003891 / 0.000200 (0.003692) | 0.000108 / 0.000054 (0.000053) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034671 / 0.037411 (-0.002740) | 0.103165 / 0.014526 (0.088639) | 0.115813 / 0.176557 (-0.060744) | 0.177407 / 0.737135 (-0.559728) | 0.117733 / 0.296338 (-0.178606) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.476859 / 0.215209 (0.261650) | 4.823063 / 2.077655 (2.745409) | 2.524133 / 1.504120 (1.020013) | 2.374482 / 1.541195 (0.833288) | 2.518047 / 1.468490 (1.049557) | 0.559131 / 4.584777 (-4.025646) | 4.126213 / 3.745712 (0.380501) | 6.488570 / 5.269862 (1.218708) | 3.816540 / 4.565676 (-0.749137) | 0.064742 / 0.424275 (-0.359533) | 0.008476 / 0.007607 (0.000869) | 0.576432 / 0.226044 (0.350387) | 5.835133 / 2.268929 (3.566205) | 3.237833 / 55.444624 (-52.206791) | 2.726596 / 6.876477 (-4.149880) | 2.799212 / 2.142072 (0.657139) | 0.661628 / 4.805227 (-4.143599) | 0.153997 / 6.500664 (-6.346667) | 0.070621 / 0.075469 (-0.004848) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.648505 / 1.841788 (-0.193282) | 22.454019 / 8.074308 (14.379711) | 16.077098 / 10.191392 (5.885706) | 0.217875 / 0.680424 (-0.462549) | 0.021285 / 0.534201 (-0.512916) | 0.459837 / 0.579283 (-0.119446) | 0.476211 / 0.434364 (0.041847) | 0.525903 / 0.540337 (-0.014435) | 0.717224 / 1.386936 (-0.669712) |

![](https://cml.dev/watermark.png#b767e9c3ef30f9da30d47cfcaccf9a7ac2500c43 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008929 / 0.011353 (-0.002424) | 0.004188 / 0.011008 (-0.006820) | 0.097030 / 0.038508 (0.058522) | 0.071363 / 0.023109 (0.048254) | 0.333116 / 0.275898 (0.057218) | 0.371272 / 0.323480 (0.047792) | 0.006430 / 0.007986 (-0.001555) | 0.003689 / 0.004328 (-0.000639) | 0.068666 / 0.004250 (0.064416) | 0.057562 / 0.037052 (0.020510) | 0.347208 / 0.258489 (0.088719) | 0.390514 / 0.293841 (0.096673) | 0.050560 / 0.128546 (-0.077987) | 0.013372 / 0.075646 (-0.062275) | 0.311345 / 0.419271 (-0.107927) | 0.068990 / 0.043533 (0.025457) | 0.363026 / 0.255139 (0.107887) | 0.379793 / 0.283200 (0.096593) | 0.036891 / 0.141683 (-0.104792) | 1.583481 / 1.452155 (0.131327) | 1.688727 / 1.492716 (0.196011) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.209777 / 0.018006 (0.191771) | 0.507267 / 0.000490 (0.506777) | 0.003637 / 0.000200 (0.003438) | 0.000105 / 0.000054 (0.000051) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029309 / 0.037411 (-0.008102) | 0.088386 / 0.014526 (0.073861) | 0.104974 / 0.176557 (-0.071582) | 0.171999 / 0.737135 (-0.565137) | 0.110797 / 0.296338 (-0.185542) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.543465 / 0.215209 (0.328256) | 5.361491 / 2.077655 (3.283836) | 2.348712 / 1.504120 (0.844592) | 2.012527 / 1.541195 (0.471332) | 2.069776 / 1.468490 (0.601286) | 0.874262 / 4.584777 (-3.710515) | 4.877317 / 3.745712 (1.131605) | 5.327459 / 5.269862 (0.057597) | 3.336823 / 4.565676 (-1.228854) | 0.100456 / 0.424275 (-0.323819) | 0.008503 / 0.007607 (0.000895) | 0.692009 / 0.226044 (0.465965) | 6.912731 / 2.268929 (4.643802) | 3.110548 / 55.444624 (-52.334076) | 2.443665 / 6.876477 (-4.432811) | 2.528713 / 2.142072 (0.386641) | 1.076358 / 4.805227 (-3.728869) | 0.220352 / 6.500664 (-6.280312) | 0.080293 / 0.075469 (0.004824) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.538444 / 1.841788 (-0.303344) | 21.121221 / 8.074308 (13.046913) | 19.810609 / 10.191392 (9.619216) | 0.225406 / 0.680424 (-0.455018) | 0.026652 / 0.534201 (-0.507549) | 0.430372 / 0.579283 (-0.148911) | 0.510722 / 0.434364 (0.076358) | 0.514347 / 0.540337 (-0.025991) | 0.686050 / 1.386936 (-0.700886) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007675 / 0.011353 (-0.003678) | 0.004542 / 0.011008 (-0.006466) | 0.069655 / 0.038508 (0.031147) | 0.069338 / 0.023109 (0.046229) | 0.436505 / 0.275898 (0.160607) | 0.481806 / 0.323480 (0.158326) | 0.005315 / 0.007986 (-0.002670) | 0.004455 / 0.004328 (0.000127) | 0.072674 / 0.004250 (0.068424) | 0.058088 / 0.037052 (0.021035) | 0.445825 / 0.258489 (0.187336) | 0.501706 / 0.293841 (0.207865) | 0.047123 / 0.128546 (-0.081424) | 0.012943 / 0.075646 (-0.062703) | 0.093491 / 0.419271 (-0.325780) | 0.060169 / 0.043533 (0.016637) | 0.436530 / 0.255139 (0.181391) | 0.466873 / 0.283200 (0.183674) | 0.040453 / 0.141683 (-0.101230) | 1.586438 / 1.452155 (0.134283) | 1.671081 / 1.492716 (0.178365) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.180607 / 0.018006 (0.162601) | 0.520145 / 0.000490 (0.519655) | 0.004824 / 0.000200 (0.004624) | 0.000116 / 0.000054 (0.000061) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029308 / 0.037411 (-0.008103) | 0.093652 / 0.014526 (0.079126) | 0.102332 / 0.176557 (-0.074224) | 0.162414 / 0.737135 (-0.574721) | 0.098017 / 0.296338 (-0.198321) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.583949 / 0.215209 (0.368740) | 6.035191 / 2.077655 (3.957536) | 2.801274 / 1.504120 (1.297155) | 2.566150 / 1.541195 (1.024955) | 2.437122 / 1.468490 (0.968632) | 0.865038 / 4.584777 (-3.719739) | 4.841727 / 3.745712 (1.096015) | 4.683919 / 5.269862 (-0.585943) | 2.941240 / 4.565676 (-1.624437) | 0.104888 / 0.424275 (-0.319387) | 0.007747 / 0.007607 (0.000140) | 0.780041 / 0.226044 (0.553997) | 7.771314 / 2.268929 (5.502385) | 3.680814 / 55.444624 (-51.763811) | 2.938472 / 6.876477 (-3.938004) | 2.981740 / 2.142072 (0.839668) | 1.065411 / 4.805227 (-3.739816) | 0.222265 / 6.500664 (-6.278399) | 0.082428 / 0.075469 (0.006959) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.626774 / 1.841788 (-0.215014) | 21.618284 / 8.074308 (13.543976) | 20.596743 / 10.191392 (10.405351) | 0.240969 / 0.680424 (-0.439454) | 0.025630 / 0.534201 (-0.508570) | 0.481981 / 0.579283 (-0.097302) | 0.547914 / 0.434364 (0.113550) | 0.522296 / 0.540337 (-0.018041) | 0.729174 / 1.386936 (-0.657762) |

![](https://cml.dev/watermark.png#b8067c0262073891180869f700ebef5ac3dc5cce "CML watermark")


Request to Share/Update Dataset Viewer Code

Hi ! The huggingface/dataset-viewer code was not maintained anymore because we switched to a new dataset viewer that is deployed available for each dataset the Hugging Face website.

What are you using this old repository for ?

I think these parts are outdated:

* https://github.com/huggingface/datasets-viewer/blob/8efad8eae313a891f713469983bf4c744786f26e/run.py#L126-L131
* https://github.com/huggingface/datasets-viewer/blob/8efad8eae313a891f713469983bf4c744786f26e/run.py#L145-L150

To make the viewer work, the first one should be replaced with the following:
```python
dataset_module = datasets.load.dataset_module_factory(path)
builder_cls = datasets.load.import_main_class(dataset_module.module_path)
confs = builder_cls.BUILDER_CONFIGS
```
And the second one:
```python
dataset_module = datasets.load.dataset_module_factory(path)
builder_cls = datasets.load.import_main_class(dataset_module.module_path)
if conf:
    builder_instance = builder_cls(name=conf, cache_dir=path if path_to_datasets is not None else None)
else:
    builder_instance = builder_cls(cache_dir=path if path_to_datasets is not None else None)
```

But as @lhoestq suggested, it's better to use the `datasets-server` API nowadays to [fetch the rows](https://huggingface.co/docs/datasets-server/rows).

> The dataset viewer on the Hugging Face website is incredibly useful

@mariosasko i think @lilyorlilypad wants to run the new dataset-viewer, not the old one

> wants to run the new dataset-viewer, not the old one

Thanks for the clarification for me. I do want to run the new dataset-viewer.

It should be possible to run it locally using the HF datasets-server API (docs [here](https://huggingface.co/docs/datasets-server)) but the front end part is not open source (yet ?)

The back-end is open source though if you're interested: https://github.com/huggingface/datasets-server
It automatically converts datasets on HF to Parquet, which is the format we use to power the viewer.

the new frontend would probably be hard to open source, as is, as it's quite intertwined with the Hub's code.

However, at some point it would be amazing to have a community-driven open source implementation of a frontend to datasets-server!

Overview:
The repository (huggingface/datasets-viewer) was recently archived and when I tried to run the code, there was the error message "AttributeError: module 'datasets.load' has no attribute 'prepare_module'". I could not resolve the issue myself due to lack of documentation of that attribute.

Request:
I kindly request the sharing of the code responsible for the dataset preview functionality or help with resolving the error. The dataset viewer on the Hugging Face website is incredibly useful since it is compatible with different types of inputs. It allows users to find datasets that meet their needs more efficiently. If needed, I am willing to contribute to the project by testing, documenting, and providing feedback on the dataset viewer code.

Thank you for considering this request, and I look forward to your response.


[FR] `map` should reuse unchanged columns from the previous dataset to avoid disk usage

New feature or request

good second issue

Issues a bit more difficult than "Good First" issues

You can use the `remove_columns` parameter in `map` to avoid duplicating the columns (and save disk space) and then concatenate the original dataset with the map result:
```python
from datasets import concatenate_datasets
# dummy example
ds_new = ds.map(lambda x: {"new_col": x["col"] + 2}, remove_columns=ds.column_names)
ds_combined = concatenate_datasets([ds, ds_new], axis=1)
```

Doing this automatically is hard to implement efficiently unless we know ahead of time which existing columns will be modified by a `map` transform. We have this info when `input_columns` are specified, so I think this is the only case we can optimize.

### Feature request

Currently adding a new column with `map` will cause all the data in the dataset to be duplicated and stored/cached on the disk again. It should reuse unchanged columns.

### Motivation

This allows having datasets with different columns but sharing some basic columns. Currently, these datasets would become too expensive to store and one would need some kind of on-the-fly join; which also doesn't seem implemented.

### Your contribution

_