fixed typo in comment

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6030). All of your documentation changes will be reflected on that endpoint.

This mistake was a bit confusing, so I thought it was worth sending a PR over.


[docs] Fix link

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007039 / 0.011353 (-0.004314) | 0.004175 / 0.011008 (-0.006833) | 0.085426 / 0.038508 (0.046918) | 0.079818 / 0.023109 (0.056709) | 0.321924 / 0.275898 (0.046026) | 0.345482 / 0.323480 (0.022002) | 0.005510 / 0.007986 (-0.002475) | 0.003452 / 0.004328 (-0.000877) | 0.065158 / 0.004250 (0.060907) | 0.058843 / 0.037052 (0.021791) | 0.316280 / 0.258489 (0.057791) | 0.351666 / 0.293841 (0.057825) | 0.031190 / 0.128546 (-0.097357) | 0.008500 / 0.075646 (-0.067147) | 0.289595 / 0.419271 (-0.129676) | 0.053798 / 0.043533 (0.010265) | 0.315804 / 0.255139 (0.060665) | 0.334957 / 0.283200 (0.051757) | 0.024350 / 0.141683 (-0.117332) | 1.515753 / 1.452155 (0.063599) | 1.556215 / 1.492716 (0.063499) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.210378 / 0.018006 (0.192372) | 0.469309 / 0.000490 (0.468820) | 0.002890 / 0.000200 (0.002690) | 0.000086 / 0.000054 (0.000031) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030214 / 0.037411 (-0.007197) | 0.088492 / 0.014526 (0.073966) | 0.098684 / 0.176557 (-0.077873) | 0.156077 / 0.737135 (-0.581058) | 0.098814 / 0.296338 (-0.197525) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.404548 / 0.215209 (0.189339) | 4.026173 / 2.077655 (1.948518) | 2.043216 / 1.504120 (0.539096) | 1.880997 / 1.541195 (0.339802) | 1.975205 / 1.468490 (0.506715) | 0.489395 / 4.584777 (-4.095382) | 3.684097 / 3.745712 (-0.061615) | 5.126934 / 5.269862 (-0.142928) | 3.092153 / 4.565676 (-1.473524) | 0.057668 / 0.424275 (-0.366607) | 0.007372 / 0.007607 (-0.000235) | 0.479647 / 0.226044 (0.253603) | 4.780207 / 2.268929 (2.511278) | 2.533457 / 55.444624 (-52.911168) | 2.182126 / 6.876477 (-4.694351) | 2.431834 / 2.142072 (0.289761) | 0.591760 / 4.805227 (-4.213467) | 0.135450 / 6.500664 (-6.365214) | 0.063218 / 0.075469 (-0.012251) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.262053 / 1.841788 (-0.579734) | 20.246992 / 8.074308 (12.172684) | 14.638222 / 10.191392 (4.446830) | 0.150021 / 0.680424 (-0.530403) | 0.018680 / 0.534201 (-0.515521) | 0.395215 / 0.579283 (-0.184068) | 0.421270 / 0.434364 (-0.013094) | 0.458845 / 0.540337 (-0.081492) | 0.634488 / 1.386936 (-0.752448) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007080 / 0.011353 (-0.004273) | 0.004112 / 0.011008 (-0.006896) | 0.066426 / 0.038508 (0.027918) | 0.090088 / 0.023109 (0.066978) | 0.400191 / 0.275898 (0.124293) | 0.429614 / 0.323480 (0.106134) | 0.005428 / 0.007986 (-0.002558) | 0.003501 / 0.004328 (-0.000827) | 0.065056 / 0.004250 (0.060806) | 0.061643 / 0.037052 (0.024590) | 0.398619 / 0.258489 (0.140130) | 0.445497 / 0.293841 (0.151657) | 0.031703 / 0.128546 (-0.096843) | 0.008708 / 0.075646 (-0.066938) | 0.071561 / 0.419271 (-0.347711) | 0.050684 / 0.043533 (0.007151) | 0.385361 / 0.255139 (0.130222) | 0.409349 / 0.283200 (0.126149) | 0.027388 / 0.141683 (-0.114295) | 1.473021 / 1.452155 (0.020866) | 1.525246 / 1.492716 (0.032529) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.237710 / 0.018006 (0.219704) | 0.468719 / 0.000490 (0.468230) | 0.000385 / 0.000200 (0.000185) | 0.000054 / 0.000054 (-0.000000) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032539 / 0.037411 (-0.004872) | 0.095324 / 0.014526 (0.080798) | 0.102248 / 0.176557 (-0.074308) | 0.156096 / 0.737135 (-0.581039) | 0.103458 / 0.296338 (-0.192881) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.416226 / 0.215209 (0.201017) | 4.141044 / 2.077655 (2.063389) | 2.143732 / 1.504120 (0.639612) | 2.001020 / 1.541195 (0.459825) | 2.091194 / 1.468490 (0.622704) | 0.489977 / 4.584777 (-4.094800) | 3.579615 / 3.745712 (-0.166097) | 3.438082 / 5.269862 (-1.831780) | 2.069031 / 4.565676 (-2.496645) | 0.056994 / 0.424275 (-0.367281) | 0.007362 / 0.007607 (-0.000245) | 0.493077 / 0.226044 (0.267033) | 4.922622 / 2.268929 (2.653694) | 2.627083 / 55.444624 (-52.817541) | 2.301141 / 6.876477 (-4.575336) | 2.356794 / 2.142072 (0.214722) | 0.583792 / 4.805227 (-4.221436) | 0.133707 / 6.500664 (-6.366958) | 0.062892 / 0.075469 (-0.012577) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.364908 / 1.841788 (-0.476880) | 20.641219 / 8.074308 (12.566911) | 14.848528 / 10.191392 (4.657136) | 0.174207 / 0.680424 (-0.506217) | 0.018206 / 0.534201 (-0.515995) | 0.413742 / 0.579283 (-0.165541) | 0.419940 / 0.434364 (-0.014424) | 0.458543 / 0.540337 (-0.081794) | 0.616518 / 1.386936 (-0.770418) |

![](https://cml.dev/watermark.png#18b2202c3e7cdde05920078f01864964556427da "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006875 / 0.011353 (-0.004478) | 0.003489 / 0.011008 (-0.007519) | 0.082077 / 0.038508 (0.043569) | 0.103011 / 0.023109 (0.079902) | 0.370572 / 0.275898 (0.094674) | 0.416400 / 0.323480 (0.092920) | 0.004048 / 0.007986 (-0.003938) | 0.003563 / 0.004328 (-0.000765) | 0.062666 / 0.004250 (0.058416) | 0.063664 / 0.037052 (0.026612) | 0.374206 / 0.258489 (0.115717) | 0.425590 / 0.293841 (0.131749) | 0.028174 / 0.128546 (-0.100373) | 0.007906 / 0.075646 (-0.067741) | 0.266251 / 0.419271 (-0.153020) | 0.045923 / 0.043533 (0.002390) | 0.376746 / 0.255139 (0.121607) | 0.401950 / 0.283200 (0.118750) | 0.024628 / 0.141683 (-0.117054) | 1.441903 / 1.452155 (-0.010252) | 1.537494 / 1.492716 (0.044777) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.214696 / 0.018006 (0.196690) | 0.425626 / 0.000490 (0.425137) | 0.003370 / 0.000200 (0.003170) | 0.000071 / 0.000054 (0.000017) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023133 / 0.037411 (-0.014279) | 0.072374 / 0.014526 (0.057848) | 0.081255 / 0.176557 (-0.095301) | 0.146960 / 0.737135 (-0.590175) | 0.081748 / 0.296338 (-0.214590) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.390683 / 0.215209 (0.175473) | 3.893166 / 2.077655 (1.815511) | 1.884321 / 1.504120 (0.380201) | 1.701899 / 1.541195 (0.160704) | 1.737839 / 1.468490 (0.269349) | 0.497008 / 4.584777 (-4.087769) | 3.041211 / 3.745712 (-0.704501) | 3.519947 / 5.269862 (-1.749915) | 2.015085 / 4.565676 (-2.550592) | 0.057685 / 0.424275 (-0.366590) | 0.006415 / 0.007607 (-0.001192) | 0.465565 / 0.226044 (0.239520) | 4.635224 / 2.268929 (2.366295) | 2.297941 / 55.444624 (-53.146683) | 1.946670 / 6.876477 (-4.929807) | 2.078527 / 2.142072 (-0.063546) | 0.584101 / 4.805227 (-4.221126) | 0.126488 / 6.500664 (-6.374176) | 0.060819 / 0.075469 (-0.014650) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.223400 / 1.841788 (-0.618388) | 17.960923 / 8.074308 (9.886615) | 13.187683 / 10.191392 (2.996291) | 0.129258 / 0.680424 (-0.551166) | 0.016601 / 0.534201 (-0.517600) | 0.330028 / 0.579283 (-0.249255) | 0.353861 / 0.434364 (-0.080503) | 0.376022 / 0.540337 (-0.164315) | 0.518145 / 1.386936 (-0.868791) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006015 / 0.011353 (-0.005338) | 0.003605 / 0.011008 (-0.007403) | 0.062169 / 0.038508 (0.023661) | 0.056094 / 0.023109 (0.032985) | 0.353085 / 0.275898 (0.077187) | 0.393744 / 0.323480 (0.070265) | 0.004672 / 0.007986 (-0.003313) | 0.002859 / 0.004328 (-0.001469) | 0.062992 / 0.004250 (0.058742) | 0.049767 / 0.037052 (0.012714) | 0.356850 / 0.258489 (0.098361) | 0.403731 / 0.293841 (0.109890) | 0.026664 / 0.128546 (-0.101882) | 0.008026 / 0.075646 (-0.067621) | 0.067944 / 0.419271 (-0.351327) | 0.042133 / 0.043533 (-0.001400) | 0.353865 / 0.255139 (0.098726) | 0.383461 / 0.283200 (0.100261) | 0.021250 / 0.141683 (-0.120433) | 1.428102 / 1.452155 (-0.024053) | 1.481061 / 1.492716 (-0.011655) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.223552 / 0.018006 (0.205546) | 0.402390 / 0.000490 (0.401900) | 0.000721 / 0.000200 (0.000521) | 0.000059 / 0.000054 (0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025065 / 0.037411 (-0.012347) | 0.075537 / 0.014526 (0.061011) | 0.083519 / 0.176557 (-0.093037) | 0.137068 / 0.737135 (-0.600068) | 0.084165 / 0.296338 (-0.212173) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.420176 / 0.215209 (0.204967) | 4.206226 / 2.077655 (2.128571) | 2.168089 / 1.504120 (0.663969) | 1.987299 / 1.541195 (0.446104) | 2.029489 / 1.468490 (0.560999) | 0.495822 / 4.584777 (-4.088955) | 3.106580 / 3.745712 (-0.639132) | 3.833215 / 5.269862 (-1.436647) | 2.450450 / 4.565676 (-2.115226) | 0.056979 / 0.424275 (-0.367296) | 0.006514 / 0.007607 (-0.001093) | 0.503646 / 0.226044 (0.277601) | 5.035035 / 2.268929 (2.766106) | 2.608245 / 55.444624 (-52.836379) | 2.245492 / 6.876477 (-4.630985) | 2.262868 / 2.142072 (0.120795) | 0.590736 / 4.805227 (-4.214491) | 0.124637 / 6.500664 (-6.376027) | 0.061442 / 0.075469 (-0.014027) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.316736 / 1.841788 (-0.525052) | 17.948635 / 8.074308 (9.874327) | 13.752442 / 10.191392 (3.561050) | 0.144107 / 0.680424 (-0.536317) | 0.017112 / 0.534201 (-0.517089) | 0.336537 / 0.579283 (-0.242746) | 0.347832 / 0.434364 (-0.086532) | 0.392944 / 0.540337 (-0.147393) | 0.534455 / 1.386936 (-0.852481) |

![](https://cml.dev/watermark.png#406b2212263c0d33f267e35b917f410ff6b3bc00 "CML watermark")

Fixes link to the builder classes :)


Use new hffs

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6028). All of your documentation changes will be reflected on that endpoint.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006665 / 0.011353 (-0.004688) | 0.004376 / 0.011008 (-0.006633) | 0.085529 / 0.038508 (0.047021) | 0.076372 / 0.023109 (0.053263) | 0.310019 / 0.275898 (0.034121) | 0.341404 / 0.323480 (0.017924) | 0.005666 / 0.007986 (-0.002320) | 0.003763 / 0.004328 (-0.000566) | 0.064678 / 0.004250 (0.060427) | 0.059283 / 0.037052 (0.022231) | 0.316194 / 0.258489 (0.057704) | 0.349397 / 0.293841 (0.055557) | 0.031199 / 0.128546 (-0.097347) | 0.008724 / 0.075646 (-0.066923) | 0.300236 / 0.419271 (-0.119035) | 0.068872 / 0.043533 (0.025339) | 0.308521 / 0.255139 (0.053382) | 0.331292 / 0.283200 (0.048092) | 0.028236 / 0.141683 (-0.113447) | 1.501365 / 1.452155 (0.049211) | 1.554334 / 1.492716 (0.061618) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.238291 / 0.018006 (0.220285) | 0.565069 / 0.000490 (0.564580) | 0.001626 / 0.000200 (0.001426) | 0.000070 / 0.000054 (0.000015) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029777 / 0.037411 (-0.007634) | 0.082873 / 0.014526 (0.068347) | 0.099619 / 0.176557 (-0.076937) | 0.156572 / 0.737135 (-0.580563) | 0.099887 / 0.296338 (-0.196452) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.401017 / 0.215209 (0.185808) | 3.827192 / 2.077655 (1.749537) | 1.861554 / 1.504120 (0.357434) | 1.699869 / 1.541195 (0.158674) | 1.720043 / 1.468490 (0.251553) | 0.486757 / 4.584777 (-4.098020) | 3.638125 / 3.745712 (-0.107587) | 5.844959 / 5.269862 (0.575097) | 3.454901 / 4.565676 (-1.110775) | 0.057650 / 0.424275 (-0.366625) | 0.007341 / 0.007607 (-0.000266) | 0.462698 / 0.226044 (0.236654) | 4.633472 / 2.268929 (2.364544) | 2.287607 / 55.444624 (-53.157017) | 2.057318 / 6.876477 (-4.819159) | 2.203657 / 2.142072 (0.061584) | 0.598136 / 4.805227 (-4.207091) | 0.134012 / 6.500664 (-6.366653) | 0.060824 / 0.075469 (-0.014645) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.277752 / 1.841788 (-0.564036) | 20.013398 / 8.074308 (11.939089) | 14.372993 / 10.191392 (4.181601) | 0.169991 / 0.680424 (-0.510433) | 0.018344 / 0.534201 (-0.515857) | 0.396985 / 0.579283 (-0.182299) | 0.416289 / 0.434364 (-0.018075) | 0.458658 / 0.540337 (-0.081680) | 0.692980 / 1.386936 (-0.693956) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006689 / 0.011353 (-0.004664) | 0.004393 / 0.011008 (-0.006615) | 0.064069 / 0.038508 (0.025561) | 0.080717 / 0.023109 (0.057607) | 0.370090 / 0.275898 (0.094191) | 0.400432 / 0.323480 (0.076952) | 0.005613 / 0.007986 (-0.002372) | 0.003641 / 0.004328 (-0.000687) | 0.064771 / 0.004250 (0.060520) | 0.057555 / 0.037052 (0.020502) | 0.392156 / 0.258489 (0.133667) | 0.409842 / 0.293841 (0.116001) | 0.031500 / 0.128546 (-0.097047) | 0.008786 / 0.075646 (-0.066860) | 0.070342 / 0.419271 (-0.348929) | 0.048646 / 0.043533 (0.005113) | 0.360914 / 0.255139 (0.105775) | 0.387626 / 0.283200 (0.104426) | 0.022787 / 0.141683 (-0.118896) | 1.508915 / 1.452155 (0.056761) | 1.539719 / 1.492716 (0.047002) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.257985 / 0.018006 (0.239979) | 0.550990 / 0.000490 (0.550501) | 0.000407 / 0.000200 (0.000207) | 0.000057 / 0.000054 (0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030183 / 0.037411 (-0.007228) | 0.086882 / 0.014526 (0.072356) | 0.102382 / 0.176557 (-0.074175) | 0.154745 / 0.737135 (-0.582390) | 0.104008 / 0.296338 (-0.192331) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.426284 / 0.215209 (0.211075) | 4.240812 / 2.077655 (2.163158) | 2.261240 / 1.504120 (0.757120) | 2.085905 / 1.541195 (0.544710) | 2.160374 / 1.468490 (0.691883) | 0.481126 / 4.584777 (-4.103651) | 3.516234 / 3.745712 (-0.229478) | 3.325322 / 5.269862 (-1.944539) | 2.043307 / 4.565676 (-2.522369) | 0.056663 / 0.424275 (-0.367612) | 0.007786 / 0.007607 (0.000179) | 0.497614 / 0.226044 (0.271570) | 4.974529 / 2.268929 (2.705600) | 2.700018 / 55.444624 (-52.744606) | 2.393778 / 6.876477 (-4.482699) | 2.628202 / 2.142072 (0.486130) | 0.594316 / 4.805227 (-4.210911) | 0.147092 / 6.500664 (-6.353572) | 0.062207 / 0.075469 (-0.013262) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.315676 / 1.841788 (-0.526112) | 20.749251 / 8.074308 (12.674943) | 14.371553 / 10.191392 (4.180160) | 0.170249 / 0.680424 (-0.510175) | 0.018478 / 0.534201 (-0.515722) | 0.395710 / 0.579283 (-0.183573) | 0.409706 / 0.434364 (-0.024658) | 0.463454 / 0.540337 (-0.076884) | 0.615657 / 1.386936 (-0.771279) |

![](https://cml.dev/watermark.png#c5a752d8e8ca0a6ed118b024ba03c1b4a2881177 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007224 / 0.011353 (-0.004129) | 0.004506 / 0.011008 (-0.006503) | 0.096729 / 0.038508 (0.058221) | 0.082394 / 0.023109 (0.059284) | 0.390954 / 0.275898 (0.115056) | 0.416647 / 0.323480 (0.093167) | 0.005894 / 0.007986 (-0.002092) | 0.003756 / 0.004328 (-0.000572) | 0.075800 / 0.004250 (0.071549) | 0.062683 / 0.037052 (0.025631) | 0.398959 / 0.258489 (0.140470) | 0.436624 / 0.293841 (0.142783) | 0.034650 / 0.128546 (-0.093896) | 0.009655 / 0.075646 (-0.065991) | 0.315761 / 0.419271 (-0.103511) | 0.060957 / 0.043533 (0.017424) | 0.385649 / 0.255139 (0.130510) | 0.394022 / 0.283200 (0.110822) | 0.024601 / 0.141683 (-0.117082) | 1.729586 / 1.452155 (0.277431) | 1.724153 / 1.492716 (0.231437) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.207070 / 0.018006 (0.189063) | 0.466502 / 0.000490 (0.466012) | 0.010739 / 0.000200 (0.010540) | 0.000214 / 0.000054 (0.000160) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031633 / 0.037411 (-0.005779) | 0.095345 / 0.014526 (0.080819) | 0.105399 / 0.176557 (-0.071157) | 0.174173 / 0.737135 (-0.562962) | 0.104207 / 0.296338 (-0.192132) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.435312 / 0.215209 (0.220103) | 4.265600 / 2.077655 (2.187946) | 2.056500 / 1.504120 (0.552380) | 1.848023 / 1.541195 (0.306828) | 1.946156 / 1.468490 (0.477666) | 0.557788 / 4.584777 (-4.026989) | 4.070289 / 3.745712 (0.324577) | 3.608027 / 5.269862 (-1.661835) | 2.214556 / 4.565676 (-2.351121) | 0.062623 / 0.424275 (-0.361652) | 0.008083 / 0.007607 (0.000476) | 0.491782 / 0.226044 (0.265738) | 4.989963 / 2.268929 (2.721035) | 2.575867 / 55.444624 (-52.868757) | 2.208045 / 6.876477 (-4.668431) | 2.364184 / 2.142072 (0.222112) | 0.633925 / 4.805227 (-4.171302) | 0.144323 / 6.500664 (-6.356341) | 0.067505 / 0.075469 (-0.007965) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.467219 / 1.841788 (-0.374569) | 22.334967 / 8.074308 (14.260659) | 15.715747 / 10.191392 (5.524355) | 0.175443 / 0.680424 (-0.504980) | 0.026165 / 0.534201 (-0.508036) | 0.490675 / 0.579283 (-0.088608) | 0.509211 / 0.434364 (0.074847) | 0.586303 / 0.540337 (0.045965) | 0.785052 / 1.386936 (-0.601884) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007893 / 0.011353 (-0.003460) | 0.004577 / 0.011008 (-0.006431) | 0.075781 / 0.038508 (0.037273) | 0.095492 / 0.023109 (0.072382) | 0.433259 / 0.275898 (0.157361) | 0.469386 / 0.323480 (0.145906) | 0.006317 / 0.007986 (-0.001669) | 0.003708 / 0.004328 (-0.000621) | 0.074417 / 0.004250 (0.070167) | 0.068605 / 0.037052 (0.031552) | 0.448701 / 0.258489 (0.190212) | 0.469131 / 0.293841 (0.175290) | 0.036647 / 0.128546 (-0.091899) | 0.010077 / 0.075646 (-0.065570) | 0.082457 / 0.419271 (-0.336815) | 0.063255 / 0.043533 (0.019722) | 0.428144 / 0.255139 (0.173005) | 0.451872 / 0.283200 (0.168672) | 0.033953 / 0.141683 (-0.107730) | 1.781752 / 1.452155 (0.329597) | 1.869014 / 1.492716 (0.376297) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.223596 / 0.018006 (0.205590) | 0.470307 / 0.000490 (0.469818) | 0.005059 / 0.000200 (0.004859) | 0.000104 / 0.000054 (0.000049) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.038804 / 0.037411 (0.001393) | 0.117879 / 0.014526 (0.103353) | 0.140701 / 0.176557 (-0.035855) | 0.194672 / 0.737135 (-0.542463) | 0.132806 / 0.296338 (-0.163533) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.510109 / 0.215209 (0.294900) | 4.729457 / 2.077655 (2.651803) | 2.512113 / 1.504120 (1.007993) | 2.302553 / 1.541195 (0.761358) | 2.420462 / 1.468490 (0.951972) | 0.531682 / 4.584777 (-4.053095) | 4.061208 / 3.745712 (0.315496) | 3.588542 / 5.269862 (-1.681320) | 2.203187 / 4.565676 (-2.362489) | 0.065791 / 0.424275 (-0.358484) | 0.008839 / 0.007607 (0.001232) | 0.562041 / 0.226044 (0.335997) | 5.702340 / 2.268929 (3.433412) | 3.127609 / 55.444624 (-52.317015) | 2.823060 / 6.876477 (-4.053417) | 2.898675 / 2.142072 (0.756603) | 0.659589 / 4.805227 (-4.145638) | 0.148798 / 6.500664 (-6.351866) | 0.070787 / 0.075469 (-0.004682) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.478317 / 1.841788 (-0.363471) | 21.995400 / 8.074308 (13.921092) | 16.770729 / 10.191392 (6.579337) | 0.226333 / 0.680424 (-0.454091) | 0.021835 / 0.534201 (-0.512366) | 0.460373 / 0.579283 (-0.118910) | 0.479494 / 0.434364 (0.045130) | 0.529470 / 0.540337 (-0.010868) | 0.718066 / 1.386936 (-0.668870) |

![](https://cml.dev/watermark.png#9a717b8eb80b0e50b25818127f79a35e0866fb14 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007824 / 0.011353 (-0.003529) | 0.004601 / 0.011008 (-0.006407) | 0.100025 / 0.038508 (0.061517) | 0.096046 / 0.023109 (0.072936) | 0.376226 / 0.275898 (0.100328) | 0.410905 / 0.323480 (0.087425) | 0.006048 / 0.007986 (-0.001938) | 0.003817 / 0.004328 (-0.000511) | 0.076624 / 0.004250 (0.072374) | 0.066390 / 0.037052 (0.029338) | 0.380098 / 0.258489 (0.121609) | 0.413603 / 0.293841 (0.119762) | 0.036546 / 0.128546 (-0.092001) | 0.009881 / 0.075646 (-0.065765) | 0.344338 / 0.419271 (-0.074934) | 0.061882 / 0.043533 (0.018350) | 0.368568 / 0.255139 (0.113429) | 0.397133 / 0.283200 (0.113934) | 0.027255 / 0.141683 (-0.114428) | 1.795099 / 1.452155 (0.342945) | 1.852443 / 1.492716 (0.359727) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.247436 / 0.018006 (0.229430) | 0.494119 / 0.000490 (0.493629) | 0.004359 / 0.000200 (0.004159) | 0.000089 / 0.000054 (0.000035) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034765 / 0.037411 (-0.002647) | 0.104541 / 0.014526 (0.090015) | 0.113898 / 0.176557 (-0.062659) | 0.183634 / 0.737135 (-0.553501) | 0.116423 / 0.296338 (-0.179916) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.458747 / 0.215209 (0.243538) | 4.555740 / 2.077655 (2.478085) | 2.217240 / 1.504120 (0.713121) | 2.039879 / 1.541195 (0.498684) | 2.088581 / 1.468490 (0.620091) | 0.588063 / 4.584777 (-3.996714) | 4.238226 / 3.745712 (0.492514) | 4.768060 / 5.269862 (-0.501802) | 2.857117 / 4.565676 (-1.708560) | 0.068742 / 0.424275 (-0.355533) | 0.008667 / 0.007607 (0.001059) | 0.549294 / 0.226044 (0.323249) | 5.464635 / 2.268929 (3.195706) | 2.744435 / 55.444624 (-52.700189) | 2.347660 / 6.876477 (-4.528816) | 2.616816 / 2.142072 (0.474743) | 0.703701 / 4.805227 (-4.101526) | 0.159749 / 6.500664 (-6.340915) | 0.071990 / 0.075469 (-0.003479) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.486599 / 1.841788 (-0.355188) | 22.745438 / 8.074308 (14.671130) | 16.822332 / 10.191392 (6.630940) | 0.184730 / 0.680424 (-0.495694) | 0.021267 / 0.534201 (-0.512934) | 0.467108 / 0.579283 (-0.112176) | 0.472674 / 0.434364 (0.038311) | 0.548094 / 0.540337 (0.007756) | 0.735885 / 1.386936 (-0.651051) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007746 / 0.011353 (-0.003607) | 0.004585 / 0.011008 (-0.006423) | 0.076943 / 0.038508 (0.038435) | 0.087473 / 0.023109 (0.064363) | 0.480099 / 0.275898 (0.204201) | 0.495271 / 0.323480 (0.171791) | 0.006348 / 0.007986 (-0.001638) | 0.003902 / 0.004328 (-0.000426) | 0.077586 / 0.004250 (0.073335) | 0.066467 / 0.037052 (0.029415) | 0.468741 / 0.258489 (0.210252) | 0.506778 / 0.293841 (0.212937) | 0.036877 / 0.128546 (-0.091669) | 0.010102 / 0.075646 (-0.065545) | 0.084419 / 0.419271 (-0.334852) | 0.058721 / 0.043533 (0.015188) | 0.453633 / 0.255139 (0.198494) | 0.481171 / 0.283200 (0.197971) | 0.028716 / 0.141683 (-0.112967) | 1.853048 / 1.452155 (0.400893) | 1.885847 / 1.492716 (0.393130) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.192136 / 0.018006 (0.174130) | 0.484481 / 0.000490 (0.483991) | 0.002951 / 0.000200 (0.002751) | 0.000098 / 0.000054 (0.000044) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.037949 / 0.037411 (0.000538) | 0.108364 / 0.014526 (0.093838) | 0.119542 / 0.176557 (-0.057014) | 0.188542 / 0.737135 (-0.548593) | 0.122011 / 0.296338 (-0.174327) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.483135 / 0.215209 (0.267926) | 4.849715 / 2.077655 (2.772060) | 2.497736 / 1.504120 (0.993616) | 2.314243 / 1.541195 (0.773048) | 2.412739 / 1.468490 (0.944249) | 0.564137 / 4.584777 (-4.020639) | 4.242273 / 3.745712 (0.496561) | 6.337843 / 5.269862 (1.067982) | 3.923250 / 4.565676 (-0.642426) | 0.066464 / 0.424275 (-0.357811) | 0.009217 / 0.007607 (0.001610) | 0.575667 / 0.226044 (0.349623) | 5.746187 / 2.268929 (3.477258) | 3.069655 / 55.444624 (-52.374969) | 2.674798 / 6.876477 (-4.201679) | 2.956535 / 2.142072 (0.814463) | 0.701043 / 4.805227 (-4.104185) | 0.157241 / 6.500664 (-6.343423) | 0.073175 / 0.075469 (-0.002294) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.609943 / 1.841788 (-0.231844) | 23.478594 / 8.074308 (15.404286) | 17.454437 / 10.191392 (7.263045) | 0.186422 / 0.680424 (-0.494002) | 0.021703 / 0.534201 (-0.512498) | 0.471704 / 0.579283 (-0.107579) | 0.480553 / 0.434364 (0.046189) | 0.552881 / 0.540337 (0.012544) | 0.722515 / 1.386936 (-0.664421) |

![](https://cml.dev/watermark.png#84645f80049cd00d9e0d4908faf3c3203fdcf21d "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007542 / 0.011353 (-0.003811) | 0.004692 / 0.011008 (-0.006316) | 0.099155 / 0.038508 (0.060647) | 0.089365 / 0.023109 (0.066256) | 0.370870 / 0.275898 (0.094972) | 0.422152 / 0.323480 (0.098673) | 0.006223 / 0.007986 (-0.001763) | 0.003852 / 0.004328 (-0.000476) | 0.075438 / 0.004250 (0.071188) | 0.065973 / 0.037052 (0.028921) | 0.381513 / 0.258489 (0.123024) | 0.416196 / 0.293841 (0.122355) | 0.035483 / 0.128546 (-0.093063) | 0.009884 / 0.075646 (-0.065762) | 0.341290 / 0.419271 (-0.077982) | 0.060546 / 0.043533 (0.017014) | 0.365101 / 0.255139 (0.109962) | 0.391058 / 0.283200 (0.107859) | 0.026325 / 0.141683 (-0.115358) | 1.815168 / 1.452155 (0.363013) | 1.834711 / 1.492716 (0.341994) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.222177 / 0.018006 (0.204171) | 0.501151 / 0.000490 (0.500662) | 0.010202 / 0.000200 (0.010002) | 0.000102 / 0.000054 (0.000048) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034043 / 0.037411 (-0.003368) | 0.097884 / 0.014526 (0.083358) | 0.114022 / 0.176557 (-0.062534) | 0.186200 / 0.737135 (-0.550935) | 0.115555 / 0.296338 (-0.180783) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.485857 / 0.215209 (0.270648) | 4.959263 / 2.077655 (2.881608) | 2.501085 / 1.504120 (0.996965) | 2.234660 / 1.541195 (0.693465) | 2.238585 / 1.468490 (0.770095) | 0.645431 / 4.584777 (-3.939345) | 4.434311 / 3.745712 (0.688599) | 4.771491 / 5.269862 (-0.498371) | 2.778963 / 4.565676 (-1.786714) | 0.075615 / 0.424275 (-0.348660) | 0.009502 / 0.007607 (0.001895) | 0.546539 / 0.226044 (0.320495) | 5.464242 / 2.268929 (3.195314) | 2.894101 / 55.444624 (-52.550524) | 2.513761 / 6.876477 (-4.362715) | 2.719843 / 2.142072 (0.577770) | 0.678828 / 4.805227 (-4.126399) | 0.157839 / 6.500664 (-6.342825) | 0.071305 / 0.075469 (-0.004164) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.496879 / 1.841788 (-0.344909) | 22.214452 / 8.074308 (14.140144) | 17.707541 / 10.191392 (7.516149) | 0.197008 / 0.680424 (-0.483416) | 0.024883 / 0.534201 (-0.509318) | 0.493611 / 0.579283 (-0.085672) | 0.500677 / 0.434364 (0.066313) | 0.569381 / 0.540337 (0.029044) | 0.773950 / 1.386936 (-0.612986) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007337 / 0.011353 (-0.004015) | 0.004572 / 0.011008 (-0.006436) | 0.091123 / 0.038508 (0.052615) | 0.079762 / 0.023109 (0.056652) | 0.450527 / 0.275898 (0.174629) | 0.525097 / 0.323480 (0.201617) | 0.005873 / 0.007986 (-0.002112) | 0.003797 / 0.004328 (-0.000532) | 0.076259 / 0.004250 (0.072009) | 0.062745 / 0.037052 (0.025692) | 0.465553 / 0.258489 (0.207064) | 0.546026 / 0.293841 (0.252186) | 0.035638 / 0.128546 (-0.092909) | 0.010086 / 0.075646 (-0.065560) | 0.109269 / 0.419271 (-0.310002) | 0.056765 / 0.043533 (0.013233) | 0.440887 / 0.255139 (0.185748) | 0.513325 / 0.283200 (0.230125) | 0.027206 / 0.141683 (-0.114476) | 1.863564 / 1.452155 (0.411409) | 1.918206 / 1.492716 (0.425490) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.266479 / 0.018006 (0.248473) | 0.487971 / 0.000490 (0.487481) | 0.012246 / 0.000200 (0.012046) | 0.000119 / 0.000054 (0.000065) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.035281 / 0.037411 (-0.002130) | 0.102991 / 0.014526 (0.088465) | 0.114638 / 0.176557 (-0.061919) | 0.184117 / 0.737135 (-0.553018) | 0.117943 / 0.296338 (-0.178396) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.497897 / 0.215209 (0.282688) | 4.973806 / 2.077655 (2.896151) | 2.596146 / 1.504120 (1.092026) | 2.419694 / 1.541195 (0.878499) | 2.525784 / 1.468490 (1.057294) | 0.568021 / 4.584777 (-4.016756) | 4.296431 / 3.745712 (0.550719) | 3.690682 / 5.269862 (-1.579179) | 2.345965 / 4.565676 (-2.219712) | 0.066859 / 0.424275 (-0.357416) | 0.009093 / 0.007607 (0.001486) | 0.582616 / 0.226044 (0.356571) | 5.826528 / 2.268929 (3.557600) | 3.253222 / 55.444624 (-52.191403) | 2.798447 / 6.876477 (-4.078030) | 3.054609 / 2.142072 (0.912537) | 0.678816 / 4.805227 (-4.126411) | 0.157966 / 6.500664 (-6.342698) | 0.073797 / 0.075469 (-0.001672) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.599480 / 1.841788 (-0.242308) | 23.249738 / 8.074308 (15.175430) | 16.965406 / 10.191392 (6.774014) | 0.171390 / 0.680424 (-0.509034) | 0.021810 / 0.534201 (-0.512391) | 0.483339 / 0.579283 (-0.095944) | 0.496615 / 0.434364 (0.062251) | 0.583786 / 0.540337 (0.043448) | 0.741699 / 1.386936 (-0.645237) |

![](https://cml.dev/watermark.png#7935cd2e564f5d1c66ed1acf731703724ba7a287 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006054 / 0.011353 (-0.005299) | 0.003706 / 0.011008 (-0.007302) | 0.080060 / 0.038508 (0.041552) | 0.061479 / 0.023109 (0.038370) | 0.327981 / 0.275898 (0.052083) | 0.356930 / 0.323480 (0.033450) | 0.004671 / 0.007986 (-0.003315) | 0.002901 / 0.004328 (-0.001428) | 0.062425 / 0.004250 (0.058174) | 0.046310 / 0.037052 (0.009258) | 0.323657 / 0.258489 (0.065168) | 0.370130 / 0.293841 (0.076289) | 0.027151 / 0.128546 (-0.101395) | 0.007850 / 0.075646 (-0.067797) | 0.262300 / 0.419271 (-0.156971) | 0.045456 / 0.043533 (0.001923) | 0.325569 / 0.255139 (0.070430) | 0.352962 / 0.283200 (0.069762) | 0.020156 / 0.141683 (-0.121527) | 1.429404 / 1.452155 (-0.022750) | 1.615032 / 1.492716 (0.122316) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.187309 / 0.018006 (0.169303) | 0.428848 / 0.000490 (0.428358) | 0.003599 / 0.000200 (0.003399) | 0.000069 / 0.000054 (0.000015) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023260 / 0.037411 (-0.014151) | 0.072467 / 0.014526 (0.057941) | 0.082398 / 0.176557 (-0.094159) | 0.142573 / 0.737135 (-0.594562) | 0.082570 / 0.296338 (-0.213768) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.426503 / 0.215209 (0.211294) | 4.267875 / 2.077655 (2.190220) | 2.189762 / 1.504120 (0.685642) | 2.027992 / 1.541195 (0.486798) | 2.053211 / 1.468490 (0.584721) | 0.503850 / 4.584777 (-4.080927) | 3.086444 / 3.745712 (-0.659268) | 3.319492 / 5.269862 (-1.950370) | 2.070714 / 4.565676 (-2.494962) | 0.057591 / 0.424275 (-0.366684) | 0.006407 / 0.007607 (-0.001200) | 0.501145 / 0.226044 (0.275100) | 5.017753 / 2.268929 (2.748825) | 2.643145 / 55.444624 (-52.801479) | 2.327440 / 6.876477 (-4.549037) | 2.460250 / 2.142072 (0.318178) | 0.589397 / 4.805227 (-4.215830) | 0.124948 / 6.500664 (-6.375716) | 0.060450 / 0.075469 (-0.015020) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.279870 / 1.841788 (-0.561918) | 18.115908 / 8.074308 (10.041600) | 13.570032 / 10.191392 (3.378640) | 0.132981 / 0.680424 (-0.547442) | 0.016942 / 0.534201 (-0.517259) | 0.333591 / 0.579283 (-0.245692) | 0.358844 / 0.434364 (-0.075520) | 0.395748 / 0.540337 (-0.144590) | 0.546213 / 1.386936 (-0.840723) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006062 / 0.011353 (-0.005291) | 0.003673 / 0.011008 (-0.007336) | 0.064726 / 0.038508 (0.026218) | 0.061854 / 0.023109 (0.038745) | 0.385343 / 0.275898 (0.109445) | 0.441284 / 0.323480 (0.117805) | 0.004830 / 0.007986 (-0.003156) | 0.002909 / 0.004328 (-0.001420) | 0.063874 / 0.004250 (0.059624) | 0.049331 / 0.037052 (0.012278) | 0.418484 / 0.258489 (0.159995) | 0.451397 / 0.293841 (0.157556) | 0.027665 / 0.128546 (-0.100881) | 0.008088 / 0.075646 (-0.067558) | 0.069625 / 0.419271 (-0.349646) | 0.043437 / 0.043533 (-0.000095) | 0.359789 / 0.255139 (0.104650) | 0.430206 / 0.283200 (0.147007) | 0.022308 / 0.141683 (-0.119375) | 1.461030 / 1.452155 (0.008875) | 1.513683 / 1.492716 (0.020966) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.230958 / 0.018006 (0.212952) | 0.417553 / 0.000490 (0.417063) | 0.000802 / 0.000200 (0.000602) | 0.000066 / 0.000054 (0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025421 / 0.037411 (-0.011991) | 0.077156 / 0.014526 (0.062630) | 0.087533 / 0.176557 (-0.089024) | 0.138048 / 0.737135 (-0.599087) | 0.089358 / 0.296338 (-0.206981) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.439172 / 0.215209 (0.223963) | 4.409509 / 2.077655 (2.331854) | 2.491270 / 1.504120 (0.987150) | 2.308446 / 1.541195 (0.767252) | 2.378440 / 1.468490 (0.909950) | 0.499834 / 4.584777 (-4.084943) | 3.083168 / 3.745712 (-0.662544) | 2.867543 / 5.269862 (-2.402318) | 1.876354 / 4.565676 (-2.689323) | 0.057092 / 0.424275 (-0.367183) | 0.006955 / 0.007607 (-0.000653) | 0.513799 / 0.226044 (0.287754) | 5.126660 / 2.268929 (2.857731) | 2.917348 / 55.444624 (-52.527277) | 2.508035 / 6.876477 (-4.368441) | 2.698089 / 2.142072 (0.556016) | 0.586828 / 4.805227 (-4.218399) | 0.124740 / 6.500664 (-6.375924) | 0.062276 / 0.075469 (-0.013193) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.291624 / 1.841788 (-0.550164) | 18.199968 / 8.074308 (10.125660) | 13.888139 / 10.191392 (3.696747) | 0.162955 / 0.680424 (-0.517469) | 0.017343 / 0.534201 (-0.516858) | 0.334683 / 0.579283 (-0.244600) | 0.352708 / 0.434364 (-0.081656) | 0.400629 / 0.540337 (-0.139708) | 0.539497 / 1.386936 (-0.847439) |

![](https://cml.dev/watermark.png#e7976db7fe22c6b93a869488d07b8137ea6a0db4 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007500 / 0.011353 (-0.003853) | 0.004498 / 0.011008 (-0.006510) | 0.100239 / 0.038508 (0.061731) | 0.083424 / 0.023109 (0.060315) | 0.366664 / 0.275898 (0.090766) | 0.406641 / 0.323480 (0.083161) | 0.004577 / 0.007986 (-0.003409) | 0.004809 / 0.004328 (0.000480) | 0.076898 / 0.004250 (0.072647) | 0.064021 / 0.037052 (0.026969) | 0.375836 / 0.258489 (0.117347) | 0.413008 / 0.293841 (0.119167) | 0.036010 / 0.128546 (-0.092537) | 0.009655 / 0.075646 (-0.065991) | 0.342595 / 0.419271 (-0.076677) | 0.061846 / 0.043533 (0.018313) | 0.376543 / 0.255139 (0.121404) | 0.395858 / 0.283200 (0.112659) | 0.026792 / 0.141683 (-0.114891) | 1.775569 / 1.452155 (0.323414) | 1.865077 / 1.492716 (0.372360) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.221521 / 0.018006 (0.203514) | 0.474604 / 0.000490 (0.474114) | 0.004354 / 0.000200 (0.004154) | 0.000090 / 0.000054 (0.000035) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032947 / 0.037411 (-0.004464) | 0.100454 / 0.014526 (0.085928) | 0.111955 / 0.176557 (-0.064602) | 0.179752 / 0.737135 (-0.557383) | 0.114282 / 0.296338 (-0.182056) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.458261 / 0.215209 (0.243052) | 4.563536 / 2.077655 (2.485881) | 2.231928 / 1.504120 (0.727808) | 2.036751 / 1.541195 (0.495556) | 2.170413 / 1.468490 (0.701923) | 0.570825 / 4.584777 (-4.013952) | 4.505762 / 3.745712 (0.760050) | 5.033461 / 5.269862 (-0.236401) | 2.704989 / 4.565676 (-1.860687) | 0.067011 / 0.424275 (-0.357264) | 0.008568 / 0.007607 (0.000961) | 0.545151 / 0.226044 (0.319106) | 5.438984 / 2.268929 (3.170055) | 2.771818 / 55.444624 (-52.672806) | 2.393082 / 6.876477 (-4.483395) | 2.467173 / 2.142072 (0.325101) | 0.678849 / 4.805227 (-4.126379) | 0.160480 / 6.500664 (-6.340184) | 0.073681 / 0.075469 (-0.001788) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.532272 / 1.841788 (-0.309516) | 22.548741 / 8.074308 (14.474433) | 17.091044 / 10.191392 (6.899652) | 0.172100 / 0.680424 (-0.508324) | 0.022220 / 0.534201 (-0.511981) | 0.467871 / 0.579283 (-0.111412) | 0.491135 / 0.434364 (0.056771) | 0.548433 / 0.540337 (0.008096) | 0.733340 / 1.386936 (-0.653596) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007593 / 0.011353 (-0.003760) | 0.004656 / 0.011008 (-0.006352) | 0.076940 / 0.038508 (0.038431) | 0.085183 / 0.023109 (0.062073) | 0.447178 / 0.275898 (0.171280) | 0.469545 / 0.323480 (0.146065) | 0.006023 / 0.007986 (-0.001962) | 0.003808 / 0.004328 (-0.000520) | 0.076767 / 0.004250 (0.072517) | 0.065713 / 0.037052 (0.028661) | 0.445573 / 0.258489 (0.187084) | 0.481689 / 0.293841 (0.187848) | 0.036893 / 0.128546 (-0.091654) | 0.009976 / 0.075646 (-0.065670) | 0.084443 / 0.419271 (-0.334829) | 0.058829 / 0.043533 (0.015297) | 0.429291 / 0.255139 (0.174152) | 0.454016 / 0.283200 (0.170816) | 0.027289 / 0.141683 (-0.114394) | 1.806786 / 1.452155 (0.354632) | 1.887680 / 1.492716 (0.394964) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.241012 / 0.018006 (0.223006) | 0.470629 / 0.000490 (0.470139) | 0.003213 / 0.000200 (0.003013) | 0.000107 / 0.000054 (0.000052) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.036896 / 0.037411 (-0.000515) | 0.106932 / 0.014526 (0.092406) | 0.120333 / 0.176557 (-0.056223) | 0.186271 / 0.737135 (-0.550865) | 0.121581 / 0.296338 (-0.174758) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.507782 / 0.215209 (0.292573) | 5.062932 / 2.077655 (2.985278) | 2.689539 / 1.504120 (1.185419) | 2.482978 / 1.541195 (0.941784) | 2.561320 / 1.468490 (1.092830) | 0.570664 / 4.584777 (-4.014113) | 4.346051 / 3.745712 (0.600339) | 6.479374 / 5.269862 (1.209513) | 4.096483 / 4.565676 (-0.469194) | 0.067564 / 0.424275 (-0.356711) | 0.009147 / 0.007607 (0.001540) | 0.596059 / 0.226044 (0.370015) | 5.963223 / 2.268929 (3.694295) | 3.201039 / 55.444624 (-52.243585) | 2.816581 / 6.876477 (-4.059896) | 3.047821 / 2.142072 (0.905748) | 0.687749 / 4.805227 (-4.117478) | 0.158174 / 6.500664 (-6.342490) | 0.073329 / 0.075469 (-0.002140) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.601346 / 1.841788 (-0.240441) | 23.712210 / 8.074308 (15.637902) | 16.567272 / 10.191392 (6.375880) | 0.224745 / 0.680424 (-0.455679) | 0.021662 / 0.534201 (-0.512539) | 0.471427 / 0.579283 (-0.107856) | 0.498751 / 0.434364 (0.064387) | 0.572047 / 0.540337 (0.031710) | 0.821868 / 1.386936 (-0.565068) |

![](https://cml.dev/watermark.png#34d0c9027c750adc89f3d04a6bf2e9cb95915da4 "CML watermark")

Thanks to @janineguo 's work in https://github.com/huggingface/datasets/pull/5919 which was needed to support HfFileSystem.

## Implementation details

I replaced all the from_hf_repo and from_local_or_remote in data_files.py to only use a new `from_patterns` which works for any fsspec path, including hf:// paths, https:// URLs and local paths. This simplifies the codebase since there is no logic duplication anymore when it comes to data files resolution.

I added `_prepare_path_and_storage_options` which returns the right storage_options to use given a path and a `DownloadConfig`. This is the only place where the logic depends on the filesystem type that must be used.

I also removed the `get_metadata_data_files_list ` and `get_patterns_and_data_files` functions added recently, since data files resolution is now handled using a common interface.

## Breaking changes

DataFilesList and DataFilesDict:
- use `str` paths instead of `Union[Path, Url]`
- support hf:// paths

close https://github.com/huggingface/datasets/issues/6017


Delete `task_templates` in `IterableDataset` when they are no longer valid

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008698 / 0.011353 (-0.002655) | 0.005250 / 0.011008 (-0.005758) | 0.104101 / 0.038508 (0.065593) | 0.085021 / 0.023109 (0.061912) | 0.426653 / 0.275898 (0.150755) | 0.460449 / 0.323480 (0.136969) | 0.005222 / 0.007986 (-0.002763) | 0.006280 / 0.004328 (0.001951) | 0.083458 / 0.004250 (0.079207) | 0.066132 / 0.037052 (0.029079) | 0.433416 / 0.258489 (0.174927) | 0.482718 / 0.293841 (0.188877) | 0.048872 / 0.128546 (-0.079675) | 0.013699 / 0.075646 (-0.061948) | 0.365660 / 0.419271 (-0.053611) | 0.071008 / 0.043533 (0.027475) | 0.428688 / 0.255139 (0.173549) | 0.443554 / 0.283200 (0.160354) | 0.035901 / 0.141683 (-0.105782) | 1.829296 / 1.452155 (0.377141) | 1.862351 / 1.492716 (0.369635) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.236284 / 0.018006 (0.218278) | 0.584075 / 0.000490 (0.583585) | 0.004634 / 0.000200 (0.004434) | 0.000125 / 0.000054 (0.000070) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034723 / 0.037411 (-0.002688) | 0.100989 / 0.014526 (0.086464) | 0.113722 / 0.176557 (-0.062834) | 0.187659 / 0.737135 (-0.549477) | 0.113937 / 0.296338 (-0.182401) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.587500 / 0.215209 (0.372291) | 5.847371 / 2.077655 (3.769716) | 2.599691 / 1.504120 (1.095571) | 2.246187 / 1.541195 (0.704992) | 2.419126 / 1.468490 (0.950636) | 0.847327 / 4.584777 (-3.737450) | 5.230438 / 3.745712 (1.484726) | 7.539021 / 5.269862 (2.269160) | 4.617473 / 4.565676 (0.051797) | 0.103620 / 0.424275 (-0.320655) | 0.009195 / 0.007607 (0.001588) | 0.714247 / 0.226044 (0.488203) | 7.331621 / 2.268929 (5.062693) | 3.416575 / 55.444624 (-52.028049) | 2.649467 / 6.876477 (-4.227009) | 2.928091 / 2.142072 (0.786018) | 1.002155 / 4.805227 (-3.803072) | 0.210790 / 6.500664 (-6.289874) | 0.081303 / 0.075469 (0.005834) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.655431 / 1.841788 (-0.186357) | 24.069595 / 8.074308 (15.995287) | 20.923766 / 10.191392 (10.732374) | 0.232021 / 0.680424 (-0.448403) | 0.026355 / 0.534201 (-0.507846) | 0.496830 / 0.579283 (-0.082453) | 0.582620 / 0.434364 (0.148257) | 0.551227 / 0.540337 (0.010890) | 0.756389 / 1.386936 (-0.630547) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009329 / 0.011353 (-0.002024) | 0.005045 / 0.011008 (-0.005964) | 0.082116 / 0.038508 (0.043608) | 0.082420 / 0.023109 (0.059311) | 0.502513 / 0.275898 (0.226615) | 0.526098 / 0.323480 (0.202618) | 0.007468 / 0.007986 (-0.000517) | 0.005477 / 0.004328 (0.001148) | 0.082617 / 0.004250 (0.078367) | 0.070292 / 0.037052 (0.033239) | 0.503290 / 0.258489 (0.244801) | 0.541631 / 0.293841 (0.247790) | 0.050826 / 0.128546 (-0.077721) | 0.014699 / 0.075646 (-0.060948) | 0.094441 / 0.419271 (-0.324830) | 0.065034 / 0.043533 (0.021501) | 0.486778 / 0.255139 (0.231639) | 0.516907 / 0.283200 (0.233707) | 0.045140 / 0.141683 (-0.096543) | 1.831676 / 1.452155 (0.379521) | 1.910865 / 1.492716 (0.418149) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.286818 / 0.018006 (0.268812) | 0.558621 / 0.000490 (0.558131) | 0.002830 / 0.000200 (0.002630) | 0.000148 / 0.000054 (0.000094) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.036716 / 0.037411 (-0.000696) | 0.107830 / 0.014526 (0.093305) | 0.116368 / 0.176557 (-0.060188) | 0.178401 / 0.737135 (-0.558734) | 0.124729 / 0.296338 (-0.171609) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.633557 / 0.215209 (0.418348) | 6.423135 / 2.077655 (4.345480) | 2.981883 / 1.504120 (1.477763) | 2.755592 / 1.541195 (1.214398) | 2.769337 / 1.468490 (1.300847) | 0.836219 / 4.584777 (-3.748558) | 5.302030 / 3.745712 (1.556318) | 7.463960 / 5.269862 (2.194098) | 4.427254 / 4.565676 (-0.138422) | 0.095990 / 0.424275 (-0.328285) | 0.009264 / 0.007607 (0.001657) | 0.770642 / 0.226044 (0.544597) | 7.779667 / 2.268929 (5.510739) | 3.799115 / 55.444624 (-51.645509) | 3.212560 / 6.876477 (-3.663917) | 3.281657 / 2.142072 (1.139584) | 1.044981 / 4.805227 (-3.760246) | 0.210693 / 6.500664 (-6.289971) | 0.079466 / 0.075469 (0.003997) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.793155 / 1.841788 (-0.048632) | 24.691127 / 8.074308 (16.616819) | 22.083150 / 10.191392 (11.891758) | 0.242246 / 0.680424 (-0.438178) | 0.028001 / 0.534201 (-0.506200) | 0.494061 / 0.579283 (-0.085222) | 0.599288 / 0.434364 (0.164924) | 0.552101 / 0.540337 (0.011764) | 0.784093 / 1.386936 (-0.602843) |

![](https://cml.dev/watermark.png#cd429c39604af34bc3a3ba1f463329b23fcbc1e3 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006658 / 0.011353 (-0.004695) | 0.004044 / 0.011008 (-0.006965) | 0.085844 / 0.038508 (0.047336) | 0.077147 / 0.023109 (0.054038) | 0.344387 / 0.275898 (0.068489) | 0.376718 / 0.323480 (0.053238) | 0.005537 / 0.007986 (-0.002448) | 0.003452 / 0.004328 (-0.000876) | 0.065326 / 0.004250 (0.061076) | 0.057639 / 0.037052 (0.020587) | 0.352363 / 0.258489 (0.093873) | 0.378939 / 0.293841 (0.085098) | 0.031259 / 0.128546 (-0.097287) | 0.008464 / 0.075646 (-0.067183) | 0.289076 / 0.419271 (-0.130195) | 0.052991 / 0.043533 (0.009459) | 0.346053 / 0.255139 (0.090914) | 0.362761 / 0.283200 (0.079561) | 0.023501 / 0.141683 (-0.118182) | 1.478312 / 1.452155 (0.026157) | 1.545437 / 1.492716 (0.052721) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.202964 / 0.018006 (0.184957) | 0.534793 / 0.000490 (0.534303) | 0.006025 / 0.000200 (0.005825) | 0.000225 / 0.000054 (0.000171) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029418 / 0.037411 (-0.007993) | 0.084297 / 0.014526 (0.069771) | 0.096702 / 0.176557 (-0.079855) | 0.157355 / 0.737135 (-0.579781) | 0.097858 / 0.296338 (-0.198480) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.380728 / 0.215209 (0.165519) | 3.787712 / 2.077655 (1.710057) | 1.836393 / 1.504120 (0.332273) | 1.678415 / 1.541195 (0.137220) | 1.781800 / 1.468490 (0.313310) | 0.478677 / 4.584777 (-4.106100) | 3.614080 / 3.745712 (-0.131632) | 3.255637 / 5.269862 (-2.014225) | 2.063642 / 4.565676 (-2.502035) | 0.056470 / 0.424275 (-0.367805) | 0.007408 / 0.007607 (-0.000199) | 0.459155 / 0.226044 (0.233111) | 4.586679 / 2.268929 (2.317750) | 2.305737 / 55.444624 (-53.138888) | 1.954755 / 6.876477 (-4.921721) | 2.190809 / 2.142072 (0.048737) | 0.572426 / 4.805227 (-4.232802) | 0.130349 / 6.500664 (-6.370315) | 0.059346 / 0.075469 (-0.016124) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.253671 / 1.841788 (-0.588117) | 19.509015 / 8.074308 (11.434706) | 13.951349 / 10.191392 (3.759957) | 0.171038 / 0.680424 (-0.509386) | 0.018826 / 0.534201 (-0.515375) | 0.394642 / 0.579283 (-0.184642) | 0.419614 / 0.434364 (-0.014750) | 0.470931 / 0.540337 (-0.069406) | 0.643858 / 1.386936 (-0.743078) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006765 / 0.011353 (-0.004587) | 0.003955 / 0.011008 (-0.007053) | 0.064377 / 0.038508 (0.025869) | 0.076980 / 0.023109 (0.053871) | 0.368675 / 0.275898 (0.092777) | 0.403746 / 0.323480 (0.080267) | 0.005303 / 0.007986 (-0.002683) | 0.003257 / 0.004328 (-0.001072) | 0.064154 / 0.004250 (0.059903) | 0.056975 / 0.037052 (0.019923) | 0.376718 / 0.258489 (0.118229) | 0.416291 / 0.293841 (0.122450) | 0.031444 / 0.128546 (-0.097102) | 0.008532 / 0.075646 (-0.067115) | 0.070455 / 0.419271 (-0.348816) | 0.049032 / 0.043533 (0.005499) | 0.361413 / 0.255139 (0.106274) | 0.384648 / 0.283200 (0.101448) | 0.024050 / 0.141683 (-0.117633) | 1.514330 / 1.452155 (0.062176) | 1.585424 / 1.492716 (0.092708) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.214701 / 0.018006 (0.196695) | 0.447706 / 0.000490 (0.447216) | 0.000373 / 0.000200 (0.000173) | 0.000058 / 0.000054 (0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031007 / 0.037411 (-0.006404) | 0.090545 / 0.014526 (0.076019) | 0.100611 / 0.176557 (-0.075945) | 0.154847 / 0.737135 (-0.582289) | 0.102864 / 0.296338 (-0.193475) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.427740 / 0.215209 (0.212531) | 4.273143 / 2.077655 (2.195488) | 2.294906 / 1.504120 (0.790786) | 2.138460 / 1.541195 (0.597265) | 2.274126 / 1.468490 (0.805636) | 0.486559 / 4.584777 (-4.098218) | 3.565554 / 3.745712 (-0.180158) | 3.377659 / 5.269862 (-1.892202) | 2.029883 / 4.565676 (-2.535793) | 0.057303 / 0.424275 (-0.366972) | 0.007314 / 0.007607 (-0.000293) | 0.504263 / 0.226044 (0.278219) | 5.041196 / 2.268929 (2.772268) | 2.819273 / 55.444624 (-52.625351) | 2.421479 / 6.876477 (-4.454998) | 2.503063 / 2.142072 (0.360991) | 0.581467 / 4.805227 (-4.223760) | 0.133532 / 6.500664 (-6.367132) | 0.062504 / 0.075469 (-0.012965) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.328765 / 1.841788 (-0.513022) | 20.131672 / 8.074308 (12.057363) | 14.312895 / 10.191392 (4.121503) | 0.191199 / 0.680424 (-0.489225) | 0.018522 / 0.534201 (-0.515679) | 0.393121 / 0.579283 (-0.186162) | 0.413122 / 0.434364 (-0.021242) | 0.469312 / 0.540337 (-0.071026) | 0.633140 / 1.386936 (-0.753796) |

![](https://cml.dev/watermark.png#dbf6c103f5844de40431478e7e4a64fbf2c2c067 "CML watermark")

Fix #6025