Fix style with ruff 0.0.278

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6026). All of your documentation changes will be reflected on that endpoint.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006444 / 0.011353 (-0.004909) | 0.003768 / 0.011008 (-0.007240) | 0.079625 / 0.038508 (0.041117) | 0.064490 / 0.023109 (0.041381) | 0.313858 / 0.275898 (0.037960) | 0.350810 / 0.323480 (0.027330) | 0.004804 / 0.007986 (-0.003182) | 0.002904 / 0.004328 (-0.001425) | 0.061728 / 0.004250 (0.057477) | 0.052265 / 0.037052 (0.015213) | 0.321246 / 0.258489 (0.062757) | 0.353873 / 0.293841 (0.060032) | 0.027510 / 0.128546 (-0.101036) | 0.007942 / 0.075646 (-0.067704) | 0.260518 / 0.419271 (-0.158754) | 0.045686 / 0.043533 (0.002153) | 0.316821 / 0.255139 (0.061682) | 0.337086 / 0.283200 (0.053886) | 0.022188 / 0.141683 (-0.119495) | 1.427345 / 1.452155 (-0.024810) | 1.476059 / 1.492716 (-0.016657) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.189640 / 0.018006 (0.171634) | 0.429724 / 0.000490 (0.429235) | 0.005314 / 0.000200 (0.005114) | 0.000076 / 0.000054 (0.000021) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024412 / 0.037411 (-0.013000) | 0.073488 / 0.014526 (0.058962) | 0.083843 / 0.176557 (-0.092714) | 0.147849 / 0.737135 (-0.589286) | 0.085465 / 0.296338 (-0.210873) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.405314 / 0.215209 (0.190105) | 4.071471 / 2.077655 (1.993816) | 1.916252 / 1.504120 (0.412132) | 1.721616 / 1.541195 (0.180422) | 1.807187 / 1.468490 (0.338697) | 0.498045 / 4.584777 (-4.086732) | 3.057526 / 3.745712 (-0.688187) | 4.451424 / 5.269862 (-0.818437) | 2.764020 / 4.565676 (-1.801656) | 0.057665 / 0.424275 (-0.366610) | 0.006679 / 0.007607 (-0.000928) | 0.485733 / 0.226044 (0.259688) | 4.844367 / 2.268929 (2.575438) | 2.435359 / 55.444624 (-53.009265) | 2.111478 / 6.876477 (-4.764999) | 2.377448 / 2.142072 (0.235375) | 0.587997 / 4.805227 (-4.217230) | 0.125545 / 6.500664 (-6.375120) | 0.061509 / 0.075469 (-0.013960) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.229210 / 1.841788 (-0.612577) | 18.553994 / 8.074308 (10.479686) | 14.037877 / 10.191392 (3.846485) | 0.144230 / 0.680424 (-0.536194) | 0.016891 / 0.534201 (-0.517310) | 0.329039 / 0.579283 (-0.250244) | 0.357269 / 0.434364 (-0.077095) | 0.384222 / 0.540337 (-0.156115) | 0.521292 / 1.386936 (-0.865644) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006359 / 0.011353 (-0.004994) | 0.003721 / 0.011008 (-0.007287) | 0.062047 / 0.038508 (0.023539) | 0.065267 / 0.023109 (0.042158) | 0.360164 / 0.275898 (0.084266) | 0.402292 / 0.323480 (0.078812) | 0.005603 / 0.007986 (-0.002382) | 0.002966 / 0.004328 (-0.001363) | 0.062580 / 0.004250 (0.058330) | 0.053634 / 0.037052 (0.016582) | 0.362210 / 0.258489 (0.103721) | 0.404285 / 0.293841 (0.110444) | 0.027567 / 0.128546 (-0.100979) | 0.008119 / 0.075646 (-0.067528) | 0.067577 / 0.419271 (-0.351694) | 0.042867 / 0.043533 (-0.000666) | 0.361576 / 0.255139 (0.106437) | 0.389061 / 0.283200 (0.105862) | 0.021923 / 0.141683 (-0.119760) | 1.446259 / 1.452155 (-0.005895) | 1.490724 / 1.492716 (-0.001992) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.206433 / 0.018006 (0.188427) | 0.424178 / 0.000490 (0.423688) | 0.002340 / 0.000200 (0.002140) | 0.000069 / 0.000054 (0.000015) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024955 / 0.037411 (-0.012456) | 0.077446 / 0.014526 (0.062920) | 0.088540 / 0.176557 (-0.088017) | 0.141225 / 0.737135 (-0.595910) | 0.089747 / 0.296338 (-0.206592) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.443738 / 0.215209 (0.228529) | 4.208887 / 2.077655 (2.131233) | 2.155127 / 1.504120 (0.651007) | 2.028178 / 1.541195 (0.486983) | 2.084903 / 1.468490 (0.616413) | 0.497530 / 4.584777 (-4.087247) | 3.069012 / 3.745712 (-0.676700) | 3.025184 / 5.269862 (-2.244678) | 1.904687 / 4.565676 (-2.660990) | 0.057526 / 0.424275 (-0.366749) | 0.006482 / 0.007607 (-0.001125) | 0.494692 / 0.226044 (0.268647) | 4.944437 / 2.268929 (2.675508) | 2.655989 / 55.444624 (-52.788635) | 2.331677 / 6.876477 (-4.544800) | 2.382396 / 2.142072 (0.240324) | 0.582019 / 4.805227 (-4.223209) | 0.125866 / 6.500664 (-6.374799) | 0.062908 / 0.075469 (-0.012561) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.294612 / 1.841788 (-0.547176) | 19.016152 / 8.074308 (10.941844) | 14.088828 / 10.191392 (3.897436) | 0.160842 / 0.680424 (-0.519582) | 0.017054 / 0.534201 (-0.517146) | 0.333647 / 0.579283 (-0.245636) | 0.348094 / 0.434364 (-0.086270) | 0.394970 / 0.540337 (-0.145367) | 0.551141 / 1.386936 (-0.835795) |

![](https://cml.dev/watermark.png#9e9cfe886792b30b5000808072a0f91ec8536749 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007442 / 0.011353 (-0.003911) | 0.004302 / 0.011008 (-0.006707) | 0.087159 / 0.038508 (0.048651) | 0.095094 / 0.023109 (0.071985) | 0.315422 / 0.275898 (0.039524) | 0.346672 / 0.323480 (0.023192) | 0.005811 / 0.007986 (-0.002174) | 0.003597 / 0.004328 (-0.000731) | 0.066400 / 0.004250 (0.062150) | 0.065947 / 0.037052 (0.028894) | 0.323269 / 0.258489 (0.064780) | 0.353309 / 0.293841 (0.059468) | 0.032268 / 0.128546 (-0.096278) | 0.008696 / 0.075646 (-0.066950) | 0.291486 / 0.419271 (-0.127786) | 0.054609 / 0.043533 (0.011076) | 0.321061 / 0.255139 (0.065922) | 0.336907 / 0.283200 (0.053707) | 0.027338 / 0.141683 (-0.114345) | 1.496442 / 1.452155 (0.044287) | 1.576946 / 1.492716 (0.084229) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.229140 / 0.018006 (0.211134) | 0.487500 / 0.000490 (0.487010) | 0.002425 / 0.000200 (0.002225) | 0.000089 / 0.000054 (0.000034) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029351 / 0.037411 (-0.008060) | 0.089610 / 0.014526 (0.075084) | 0.097880 / 0.176557 (-0.078676) | 0.155947 / 0.737135 (-0.581189) | 0.098593 / 0.296338 (-0.197745) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.382911 / 0.215209 (0.167702) | 3.820363 / 2.077655 (1.742708) | 1.866385 / 1.504120 (0.362265) | 1.712910 / 1.541195 (0.171716) | 1.813863 / 1.468490 (0.345373) | 0.484884 / 4.584777 (-4.099893) | 3.678911 / 3.745712 (-0.066801) | 5.249908 / 5.269862 (-0.019953) | 3.099614 / 4.565676 (-1.466063) | 0.057449 / 0.424275 (-0.366826) | 0.007728 / 0.007607 (0.000120) | 0.462123 / 0.226044 (0.236078) | 4.603942 / 2.268929 (2.335014) | 2.380957 / 55.444624 (-53.063668) | 2.059621 / 6.876477 (-4.816856) | 2.293764 / 2.142072 (0.151691) | 0.636471 / 4.805227 (-4.168756) | 0.150112 / 6.500664 (-6.350552) | 0.063705 / 0.075469 (-0.011764) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.358099 / 1.841788 (-0.483689) | 20.193750 / 8.074308 (12.119442) | 14.297350 / 10.191392 (4.105958) | 0.164477 / 0.680424 (-0.515947) | 0.018259 / 0.534201 (-0.515942) | 0.399010 / 0.579283 (-0.180273) | 0.417306 / 0.434364 (-0.017058) | 0.456961 / 0.540337 (-0.083377) | 0.631068 / 1.386936 (-0.755868) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007324 / 0.011353 (-0.004028) | 0.004463 / 0.011008 (-0.006545) | 0.066148 / 0.038508 (0.027640) | 0.093909 / 0.023109 (0.070799) | 0.399122 / 0.275898 (0.123224) | 0.430226 / 0.323480 (0.106746) | 0.005505 / 0.007986 (-0.002481) | 0.003579 / 0.004328 (-0.000749) | 0.066529 / 0.004250 (0.062278) | 0.063471 / 0.037052 (0.026418) | 0.406351 / 0.258489 (0.147862) | 0.439987 / 0.293841 (0.146146) | 0.032640 / 0.128546 (-0.095906) | 0.008770 / 0.075646 (-0.066877) | 0.072592 / 0.419271 (-0.346680) | 0.050429 / 0.043533 (0.006896) | 0.390873 / 0.255139 (0.135734) | 0.412438 / 0.283200 (0.129239) | 0.027113 / 0.141683 (-0.114570) | 1.458281 / 1.452155 (0.006126) | 1.536819 / 1.492716 (0.044103) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.228309 / 0.018006 (0.210303) | 0.454042 / 0.000490 (0.453552) | 0.000387 / 0.000200 (0.000187) | 0.000055 / 0.000054 (0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029573 / 0.037411 (-0.007838) | 0.086433 / 0.014526 (0.071907) | 0.097992 / 0.176557 (-0.078565) | 0.152464 / 0.737135 (-0.584671) | 0.099901 / 0.296338 (-0.196437) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.413807 / 0.215209 (0.198598) | 4.126395 / 2.077655 (2.048740) | 2.113544 / 1.504120 (0.609424) | 1.967829 / 1.541195 (0.426635) | 2.037123 / 1.468490 (0.568633) | 0.489403 / 4.584777 (-4.095374) | 3.689508 / 3.745712 (-0.056204) | 3.503909 / 5.269862 (-1.765952) | 2.113812 / 4.565676 (-2.451864) | 0.057988 / 0.424275 (-0.366287) | 0.007336 / 0.007607 (-0.000271) | 0.490840 / 0.226044 (0.264795) | 4.885040 / 2.268929 (2.616112) | 2.627864 / 55.444624 (-52.816760) | 2.231467 / 6.876477 (-4.645010) | 2.251307 / 2.142072 (0.109235) | 0.577370 / 4.805227 (-4.227857) | 0.131770 / 6.500664 (-6.368895) | 0.061313 / 0.075469 (-0.014156) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.362052 / 1.841788 (-0.479735) | 21.332694 / 8.074308 (13.258386) | 15.562019 / 10.191392 (5.370627) | 0.170874 / 0.680424 (-0.509550) | 0.019226 / 0.534201 (-0.514975) | 0.400311 / 0.579283 (-0.178972) | 0.423060 / 0.434364 (-0.011304) | 0.469946 / 0.540337 (-0.070391) | 0.647745 / 1.386936 (-0.739191) |

![](https://cml.dev/watermark.png#aec567c2f224f192e6e1f9799e3afc755eb517b2 "CML watermark")


Using a dataset for a use other than it was intended for.

I've opened a PR with a fix. In the meantime, you can avoid the error by deleting `task_templates` with `dataset.info.task_templates = None` before the `interleave_datasets` call.
`

### Describe the bug

Hi, I want to use the rotten tomatoes dataset but for a task other than classification, but when I interleave the dataset, it throws ```'ValueError: Column label is not present in features.'```. It seems that the label_col must be there in the dataset for some reason?

Here is the full stacktrace

```
  File "/home/suryahari/Vornoi/tryage-handoff-other-datasets.py", line 276, in create_dataloaders
    dataset = interleave_datasets(dsfold, stopping_strategy="all_exhausted")
  File "/home/suryahari/miniconda3/envs/vornoi/lib/python3.10/site-packages/datasets/combine.py", line 134, in interleave_datasets
    return _interleave_iterable_datasets(
  File "/home/suryahari/miniconda3/envs/vornoi/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 1833, in _interleave_iterable_datasets
    info = DatasetInfo.from_merge([d.info for d in datasets])
  File "/home/suryahari/miniconda3/envs/vornoi/lib/python3.10/site-packages/datasets/info.py", line 275, in from_merge
    dataset_infos = [dset_info.copy() for dset_info in dataset_infos if dset_info is not None]
  File "/home/suryahari/miniconda3/envs/vornoi/lib/python3.10/site-packages/datasets/info.py", line 275, in <listcomp>
    dataset_infos = [dset_info.copy() for dset_info in dataset_infos if dset_info is not None]
  File "/home/suryahari/miniconda3/envs/vornoi/lib/python3.10/site-packages/datasets/info.py", line 378, in copy
    return self.__class__(**{k: copy.deepcopy(v) for k, v in self.__dict__.items()})
  File "<string>", line 20, in __init__
  File "/home/suryahari/miniconda3/envs/vornoi/lib/python3.10/site-packages/datasets/info.py", line 208, in __post_init__
    self.task_templates = [
  File "/home/suryahari/miniconda3/envs/vornoi/lib/python3.10/site-packages/datasets/info.py", line 209, in <listcomp>
    template.align_with_features(self.features) for template in (self.task_templates)
  File "/home/suryahari/miniconda3/envs/vornoi/lib/python3.10/site-packages/datasets/tasks/text_classification.py", line 20, in align_with_features
    raise ValueError(f"Column {self.label_column} is not present in features.")
ValueError: Column label is not present in features.
```

### Steps to reproduce the bug

Delete the column `labels` from the `rotten_tomatoes` dataset. Try to interleave it with other datasets.

### Expected behavior

Should let me use the dataset with just the `text` field

### Environment info

latest datasets library? I don't think this was an issue in earlier versions.


Don't reference self in Spark._validate_cache_dir

Ptal @lhoestq :) I tested this manually on a multi-node Databricks cluster

Hm looks like the check_code_quality failures are unrelated to me change... https://github.com/huggingface/datasets/actions/runs/5536162850/jobs/10103451883?pr=6024

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005952 / 0.011353 (-0.005400) | 0.003585 / 0.011008 (-0.007424) | 0.079163 / 0.038508 (0.040655) | 0.057926 / 0.023109 (0.034817) | 0.326647 / 0.275898 (0.050749) | 0.383485 / 0.323480 (0.060005) | 0.004530 / 0.007986 (-0.003456) | 0.002821 / 0.004328 (-0.001508) | 0.062071 / 0.004250 (0.057820) | 0.048023 / 0.037052 (0.010971) | 0.329368 / 0.258489 (0.070879) | 0.390877 / 0.293841 (0.097036) | 0.026959 / 0.128546 (-0.101588) | 0.007911 / 0.075646 (-0.067735) | 0.259956 / 0.419271 (-0.159315) | 0.044582 / 0.043533 (0.001049) | 0.320537 / 0.255139 (0.065398) | 0.373814 / 0.283200 (0.090614) | 0.020275 / 0.141683 (-0.121408) | 1.532128 / 1.452155 (0.079973) | 1.595031 / 1.492716 (0.102315) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.186127 / 0.018006 (0.168120) | 0.428586 / 0.000490 (0.428097) | 0.005180 / 0.000200 (0.004980) | 0.000069 / 0.000054 (0.000015) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024876 / 0.037411 (-0.012536) | 0.072169 / 0.014526 (0.057643) | 0.082015 / 0.176557 (-0.094542) | 0.147467 / 0.737135 (-0.589668) | 0.082769 / 0.296338 (-0.213570) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.410625 / 0.215209 (0.195416) | 4.116742 / 2.077655 (2.039088) | 2.172291 / 1.504120 (0.668171) | 2.022462 / 1.541195 (0.481268) | 2.048142 / 1.468490 (0.579651) | 0.503152 / 4.584777 (-4.081625) | 3.019135 / 3.745712 (-0.726577) | 3.589451 / 5.269862 (-1.680410) | 2.206876 / 4.565676 (-2.358801) | 0.057687 / 0.424275 (-0.366588) | 0.006560 / 0.007607 (-0.001047) | 0.475585 / 0.226044 (0.249541) | 4.784344 / 2.268929 (2.515416) | 2.506322 / 55.444624 (-52.938302) | 2.168251 / 6.876477 (-4.708225) | 2.324453 / 2.142072 (0.182381) | 0.590609 / 4.805227 (-4.214618) | 0.124178 / 6.500664 (-6.376486) | 0.059197 / 0.075469 (-0.016272) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.212359 / 1.841788 (-0.629429) | 17.915843 / 8.074308 (9.841535) | 13.128330 / 10.191392 (2.936938) | 0.144805 / 0.680424 (-0.535618) | 0.016889 / 0.534201 (-0.517312) | 0.344056 / 0.579283 (-0.235227) | 0.359370 / 0.434364 (-0.074994) | 0.404199 / 0.540337 (-0.136138) | 0.549117 / 1.386936 (-0.837819) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005914 / 0.011353 (-0.005439) | 0.003565 / 0.011008 (-0.007443) | 0.061575 / 0.038508 (0.023067) | 0.057677 / 0.023109 (0.034568) | 0.359753 / 0.275898 (0.083855) | 0.394135 / 0.323480 (0.070655) | 0.004648 / 0.007986 (-0.003338) | 0.002795 / 0.004328 (-0.001534) | 0.061877 / 0.004250 (0.057626) | 0.049673 / 0.037052 (0.012621) | 0.363120 / 0.258489 (0.104631) | 0.402685 / 0.293841 (0.108844) | 0.027021 / 0.128546 (-0.101525) | 0.008006 / 0.075646 (-0.067641) | 0.067398 / 0.419271 (-0.351874) | 0.044442 / 0.043533 (0.000909) | 0.364851 / 0.255139 (0.109712) | 0.387219 / 0.283200 (0.104019) | 0.027267 / 0.141683 (-0.114416) | 1.466675 / 1.452155 (0.014520) | 1.512607 / 1.492716 (0.019891) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.206156 / 0.018006 (0.188150) | 0.410877 / 0.000490 (0.410387) | 0.003061 / 0.000200 (0.002861) | 0.000068 / 0.000054 (0.000013) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024869 / 0.037411 (-0.012542) | 0.075736 / 0.014526 (0.061210) | 0.083922 / 0.176557 (-0.092634) | 0.139510 / 0.737135 (-0.597626) | 0.087685 / 0.296338 (-0.208654) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.414473 / 0.215209 (0.199264) | 4.150633 / 2.077655 (2.072979) | 2.132892 / 1.504120 (0.628773) | 1.964072 / 1.541195 (0.422878) | 2.003353 / 1.468490 (0.534863) | 0.498012 / 4.584777 (-4.086765) | 3.010135 / 3.745712 (-0.735577) | 2.841130 / 5.269862 (-2.428732) | 1.826013 / 4.565676 (-2.739664) | 0.057443 / 0.424275 (-0.366832) | 0.006374 / 0.007607 (-0.001234) | 0.490337 / 0.226044 (0.264292) | 4.889628 / 2.268929 (2.620700) | 2.575626 / 55.444624 (-52.868998) | 2.246522 / 6.876477 (-4.629955) | 2.276183 / 2.142072 (0.134110) | 0.581465 / 4.805227 (-4.223763) | 0.123877 / 6.500664 (-6.376787) | 0.060339 / 0.075469 (-0.015130) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.333202 / 1.841788 (-0.508585) | 18.363558 / 8.074308 (10.289250) | 14.109356 / 10.191392 (3.917964) | 0.147358 / 0.680424 (-0.533066) | 0.016813 / 0.534201 (-0.517388) | 0.334815 / 0.579283 (-0.244468) | 0.366576 / 0.434364 (-0.067788) | 0.397223 / 0.540337 (-0.143115) | 0.547893 / 1.386936 (-0.839043) |

![](https://cml.dev/watermark.png#67ac60bcbebe9ddac70264951b1d584c93003cdf "CML watermark")

Fix for https://github.com/huggingface/datasets/issues/5963


Fix `ClassLabel` min max check for `None` values

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007108 / 0.011353 (-0.004245) | 0.004446 / 0.011008 (-0.006562) | 0.084013 / 0.038508 (0.045505) | 0.084271 / 0.023109 (0.061162) | 0.324496 / 0.275898 (0.048598) | 0.347783 / 0.323480 (0.024303) | 0.004382 / 0.007986 (-0.003604) | 0.005200 / 0.004328 (0.000872) | 0.065117 / 0.004250 (0.060866) | 0.063368 / 0.037052 (0.026316) | 0.328731 / 0.258489 (0.070242) | 0.356676 / 0.293841 (0.062835) | 0.031155 / 0.128546 (-0.097392) | 0.008672 / 0.075646 (-0.066975) | 0.287573 / 0.419271 (-0.131698) | 0.053692 / 0.043533 (0.010160) | 0.308796 / 0.255139 (0.053657) | 0.330521 / 0.283200 (0.047321) | 0.025010 / 0.141683 (-0.116672) | 1.498968 / 1.452155 (0.046813) | 1.552096 / 1.492716 (0.059380) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.263580 / 0.018006 (0.245574) | 0.559765 / 0.000490 (0.559275) | 0.003450 / 0.000200 (0.003250) | 0.000079 / 0.000054 (0.000024) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029403 / 0.037411 (-0.008008) | 0.088154 / 0.014526 (0.073628) | 0.100372 / 0.176557 (-0.076185) | 0.157777 / 0.737135 (-0.579359) | 0.102273 / 0.296338 (-0.194066) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.387027 / 0.215209 (0.171818) | 3.854260 / 2.077655 (1.776605) | 1.875159 / 1.504120 (0.371039) | 1.703734 / 1.541195 (0.162539) | 1.814305 / 1.468490 (0.345815) | 0.482524 / 4.584777 (-4.102253) | 3.463602 / 3.745712 (-0.282110) | 4.004766 / 5.269862 (-1.265095) | 2.406751 / 4.565676 (-2.158925) | 0.057069 / 0.424275 (-0.367206) | 0.007448 / 0.007607 (-0.000159) | 0.465801 / 0.226044 (0.239757) | 4.636700 / 2.268929 (2.367771) | 2.329475 / 55.444624 (-53.115150) | 1.998330 / 6.876477 (-4.878146) | 2.264617 / 2.142072 (0.122544) | 0.577998 / 4.805227 (-4.227230) | 0.130846 / 6.500664 (-6.369818) | 0.059713 / 0.075469 (-0.015756) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.275931 / 1.841788 (-0.565857) | 20.396288 / 8.074308 (12.321980) | 13.875242 / 10.191392 (3.683850) | 0.164367 / 0.680424 (-0.516057) | 0.018573 / 0.534201 (-0.515628) | 0.397516 / 0.579283 (-0.181767) | 0.398977 / 0.434364 (-0.035387) | 0.462386 / 0.540337 (-0.077951) | 0.610129 / 1.386936 (-0.776807) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006912 / 0.011353 (-0.004441) | 0.004212 / 0.011008 (-0.006797) | 0.065707 / 0.038508 (0.027199) | 0.090435 / 0.023109 (0.067325) | 0.380539 / 0.275898 (0.104641) | 0.412692 / 0.323480 (0.089212) | 0.005545 / 0.007986 (-0.002441) | 0.003657 / 0.004328 (-0.000672) | 0.065380 / 0.004250 (0.061130) | 0.062901 / 0.037052 (0.025848) | 0.385931 / 0.258489 (0.127442) | 0.416272 / 0.293841 (0.122431) | 0.031974 / 0.128546 (-0.096572) | 0.008783 / 0.075646 (-0.066863) | 0.071424 / 0.419271 (-0.347847) | 0.049454 / 0.043533 (0.005921) | 0.374231 / 0.255139 (0.119092) | 0.386530 / 0.283200 (0.103331) | 0.025404 / 0.141683 (-0.116279) | 1.469869 / 1.452155 (0.017715) | 1.548629 / 1.492716 (0.055913) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.218413 / 0.018006 (0.200406) | 0.573863 / 0.000490 (0.573373) | 0.004156 / 0.000200 (0.003956) | 0.000097 / 0.000054 (0.000043) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032610 / 0.037411 (-0.004801) | 0.088270 / 0.014526 (0.073744) | 0.106821 / 0.176557 (-0.069735) | 0.164498 / 0.737135 (-0.572638) | 0.106881 / 0.296338 (-0.189457) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.433730 / 0.215209 (0.218520) | 4.323902 / 2.077655 (2.246247) | 2.308607 / 1.504120 (0.804487) | 2.138888 / 1.541195 (0.597693) | 2.246760 / 1.468490 (0.778269) | 0.486863 / 4.584777 (-4.097914) | 3.561826 / 3.745712 (-0.183886) | 5.592685 / 5.269862 (0.322824) | 3.318560 / 4.565676 (-1.247116) | 0.057348 / 0.424275 (-0.366927) | 0.007434 / 0.007607 (-0.000174) | 0.506767 / 0.226044 (0.280723) | 5.083097 / 2.268929 (2.814168) | 2.780618 / 55.444624 (-52.664006) | 2.456924 / 6.876477 (-4.419553) | 2.564184 / 2.142072 (0.422112) | 0.580693 / 4.805227 (-4.224534) | 0.134471 / 6.500664 (-6.366194) | 0.062883 / 0.075469 (-0.012586) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.346618 / 1.841788 (-0.495169) | 20.547998 / 8.074308 (12.473690) | 14.404159 / 10.191392 (4.212767) | 0.176612 / 0.680424 (-0.503812) | 0.018372 / 0.534201 (-0.515829) | 0.395636 / 0.579283 (-0.183647) | 0.410661 / 0.434364 (-0.023703) | 0.468782 / 0.540337 (-0.071555) | 0.637476 / 1.386936 (-0.749460) |

![](https://cml.dev/watermark.png#0172d4dac0ca823e8bd293cfd4d28e78d92efe42 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009896 / 0.011353 (-0.001457) | 0.004658 / 0.011008 (-0.006351) | 0.101185 / 0.038508 (0.062677) | 0.075480 / 0.023109 (0.052371) | 0.410620 / 0.275898 (0.134722) | 0.470639 / 0.323480 (0.147159) | 0.007042 / 0.007986 (-0.000943) | 0.003909 / 0.004328 (-0.000419) | 0.079676 / 0.004250 (0.075425) | 0.066921 / 0.037052 (0.029869) | 0.423624 / 0.258489 (0.165135) | 0.473008 / 0.293841 (0.179167) | 0.048492 / 0.128546 (-0.080054) | 0.012833 / 0.075646 (-0.062813) | 0.335286 / 0.419271 (-0.083985) | 0.083506 / 0.043533 (0.039973) | 0.401918 / 0.255139 (0.146779) | 0.467975 / 0.283200 (0.184775) | 0.050025 / 0.141683 (-0.091658) | 1.679392 / 1.452155 (0.227237) | 1.852812 / 1.492716 (0.360095) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.248067 / 0.018006 (0.230061) | 0.584818 / 0.000490 (0.584328) | 0.021558 / 0.000200 (0.021358) | 0.000104 / 0.000054 (0.000050) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028572 / 0.037411 (-0.008839) | 0.097212 / 0.014526 (0.082686) | 0.121675 / 0.176557 (-0.054881) | 0.186597 / 0.737135 (-0.550538) | 0.122285 / 0.296338 (-0.174053) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.586279 / 0.215209 (0.371070) | 5.634402 / 2.077655 (3.556747) | 2.560648 / 1.504120 (1.056528) | 2.288796 / 1.541195 (0.747601) | 2.402580 / 1.468490 (0.934090) | 0.801453 / 4.584777 (-3.783324) | 5.036654 / 3.745712 (1.290942) | 8.319972 / 5.269862 (3.050110) | 4.665620 / 4.565676 (0.099944) | 0.107292 / 0.424275 (-0.316983) | 0.009206 / 0.007607 (0.001599) | 0.766505 / 0.226044 (0.540461) | 7.333784 / 2.268929 (5.064856) | 3.601875 / 55.444624 (-51.842749) | 2.886388 / 6.876477 (-3.990089) | 3.231797 / 2.142072 (1.089725) | 1.179509 / 4.805227 (-3.625718) | 0.224656 / 6.500664 (-6.276008) | 0.084749 / 0.075469 (0.009280) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.772345 / 1.841788 (-0.069443) | 24.138788 / 8.074308 (16.064480) | 20.712416 / 10.191392 (10.521024) | 0.254655 / 0.680424 (-0.425769) | 0.028858 / 0.534201 (-0.505343) | 0.499314 / 0.579283 (-0.079969) | 0.605797 / 0.434364 (0.171433) | 0.567628 / 0.540337 (0.027290) | 0.752288 / 1.386936 (-0.634648) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.010134 / 0.011353 (-0.001219) | 0.004630 / 0.011008 (-0.006378) | 0.082282 / 0.038508 (0.043774) | 0.081722 / 0.023109 (0.058613) | 0.465018 / 0.275898 (0.189120) | 0.516392 / 0.323480 (0.192912) | 0.006618 / 0.007986 (-0.001368) | 0.004310 / 0.004328 (-0.000018) | 0.078990 / 0.004250 (0.074739) | 0.077729 / 0.037052 (0.040677) | 0.464892 / 0.258489 (0.206403) | 0.510551 / 0.293841 (0.216710) | 0.050750 / 0.128546 (-0.077796) | 0.014402 / 0.075646 (-0.061244) | 0.092587 / 0.419271 (-0.326685) | 0.074769 / 0.043533 (0.031237) | 0.468591 / 0.255139 (0.213452) | 0.508138 / 0.283200 (0.224938) | 0.047774 / 0.141683 (-0.093909) | 1.798354 / 1.452155 (0.346199) | 1.851431 / 1.492716 (0.358714) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.282528 / 0.018006 (0.264522) | 0.588286 / 0.000490 (0.587797) | 0.004892 / 0.000200 (0.004692) | 0.000136 / 0.000054 (0.000082) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.037048 / 0.037411 (-0.000364) | 0.101513 / 0.014526 (0.086987) | 0.133238 / 0.176557 (-0.043319) | 0.234799 / 0.737135 (-0.502336) | 0.120636 / 0.296338 (-0.175703) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.615377 / 0.215209 (0.400168) | 6.225717 / 2.077655 (4.148062) | 2.974137 / 1.504120 (1.470018) | 2.642168 / 1.541195 (1.100973) | 2.706051 / 1.468490 (1.237561) | 0.837171 / 4.584777 (-3.747606) | 5.143368 / 3.745712 (1.397656) | 4.560241 / 5.269862 (-0.709621) | 2.838375 / 4.565676 (-1.727301) | 0.092505 / 0.424275 (-0.331770) | 0.008962 / 0.007607 (0.001355) | 0.726361 / 0.226044 (0.500317) | 7.323998 / 2.268929 (5.055070) | 3.650531 / 55.444624 (-51.794094) | 2.960886 / 6.876477 (-3.915591) | 3.003889 / 2.142072 (0.861816) | 0.979264 / 4.805227 (-3.825963) | 0.204531 / 6.500664 (-6.296133) | 0.078285 / 0.075469 (0.002816) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.774225 / 1.841788 (-0.067563) | 26.399536 / 8.074308 (18.325228) | 22.312890 / 10.191392 (12.121498) | 0.244651 / 0.680424 (-0.435773) | 0.026950 / 0.534201 (-0.507251) | 0.493037 / 0.579283 (-0.086246) | 0.620399 / 0.434364 (0.186036) | 0.748985 / 0.540337 (0.208648) | 0.799766 / 1.386936 (-0.587170) |

![](https://cml.dev/watermark.png#a49ac2864177ec4fb34c43b59a6e49de1f21f973 "CML watermark")

Fix #6022


Batch map raises TypeError: '>=' not supported between instances of 'NoneType' and 'int'

Thanks for reporting! I've opened a PR with a fix.

### Describe the bug

When mapping some datasets with `batched=True`, datasets may raise an exeception:

```python
Traceback (most recent call last):
  File "/Users/codingl2k1/Work/datasets/venv/lib/python3.11/site-packages/multiprocess/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "/Users/codingl2k1/Work/datasets/src/datasets/utils/py_utils.py", line 1328, in _write_generator_to_queue
    for i, result in enumerate(func(**kwargs)):
  File "/Users/codingl2k1/Work/datasets/src/datasets/arrow_dataset.py", line 3483, in _map_single
    writer.write_batch(batch)
  File "/Users/codingl2k1/Work/datasets/src/datasets/arrow_writer.py", line 549, in write_batch
    array = cast_array_to_feature(col_values, col_type) if col_type is not None else col_values
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/codingl2k1/Work/datasets/src/datasets/table.py", line 1831, in wrapper
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/codingl2k1/Work/datasets/src/datasets/table.py", line 1831, in <listcomp>
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/codingl2k1/Work/datasets/src/datasets/table.py", line 2063, in cast_array_to_feature
    return feature.cast_storage(array)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/codingl2k1/Work/datasets/src/datasets/features/features.py", line 1098, in cast_storage
    if min_max["max"] >= self.num_classes:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: '>=' not supported between instances of 'NoneType' and 'int'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/Users/codingl2k1/Work/datasets/t1.py", line 33, in <module>
    ds = ds.map(transforms, num_proc=14, batched=True, batch_size=5)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/codingl2k1/Work/datasets/src/datasets/dataset_dict.py", line 850, in map
    {
  File "/Users/codingl2k1/Work/datasets/src/datasets/dataset_dict.py", line 851, in <dictcomp>
    k: dataset.map(
       ^^^^^^^^^^^^
  File "/Users/codingl2k1/Work/datasets/src/datasets/arrow_dataset.py", line 577, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/codingl2k1/Work/datasets/src/datasets/arrow_dataset.py", line 542, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/codingl2k1/Work/datasets/src/datasets/arrow_dataset.py", line 3179, in map
    for rank, done, content in iflatmap_unordered(
  File "/Users/codingl2k1/Work/datasets/src/datasets/utils/py_utils.py", line 1368, in iflatmap_unordered
    [async_result.get(timeout=0.05) for async_result in async_results]
  File "/Users/codingl2k1/Work/datasets/src/datasets/utils/py_utils.py", line 1368, in <listcomp>
    [async_result.get(timeout=0.05) for async_result in async_results]
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/codingl2k1/Work/datasets/venv/lib/python3.11/site-packages/multiprocess/pool.py", line 774, in get
    raise self._value
TypeError: '>=' not supported between instances of 'NoneType' and 'int'
```

### Steps to reproduce the bug

1. Checkout the latest main of datasets.
2. Run the code:

```python
from datasets import load_dataset

def transforms(examples):
    # examples["pixel_values"] = [image.convert("RGB").resize((100, 100)) for image in examples["image"]]
    return examples

ds = load_dataset("scene_parse_150")
ds = ds.map(transforms, num_proc=14, batched=True, batch_size=5)
print(ds)
```

### Expected behavior

map without exception.

### Environment info

Datasets: https://github.com/huggingface/datasets/commit/b8067c0262073891180869f700ebef5ac3dc5cce
Python: 3.11.4
System: Macos


[docs] Update return statement of index search

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007697 / 0.011353 (-0.003656) | 0.004233 / 0.011008 (-0.006776) | 0.087890 / 0.038508 (0.049382) | 0.065305 / 0.023109 (0.042196) | 0.366919 / 0.275898 (0.091020) | 0.399656 / 0.323480 (0.076176) | 0.006753 / 0.007986 (-0.001232) | 0.003428 / 0.004328 (-0.000900) | 0.070180 / 0.004250 (0.065930) | 0.054164 / 0.037052 (0.017112) | 0.377130 / 0.258489 (0.118641) | 0.403456 / 0.293841 (0.109615) | 0.042639 / 0.128546 (-0.085907) | 0.012396 / 0.075646 (-0.063250) | 0.314235 / 0.419271 (-0.105036) | 0.061976 / 0.043533 (0.018443) | 0.376959 / 0.255139 (0.121820) | 0.433313 / 0.283200 (0.150113) | 0.031253 / 0.141683 (-0.110430) | 1.555749 / 1.452155 (0.103594) | 1.643905 / 1.492716 (0.151189) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.208630 / 0.018006 (0.190624) | 0.519532 / 0.000490 (0.519042) | 0.003719 / 0.000200 (0.003519) | 0.000099 / 0.000054 (0.000045) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027403 / 0.037411 (-0.010008) | 0.080990 / 0.014526 (0.066464) | 0.090424 / 0.176557 (-0.086133) | 0.153922 / 0.737135 (-0.583213) | 0.098156 / 0.296338 (-0.198183) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.519453 / 0.215209 (0.304244) | 5.100089 / 2.077655 (3.022434) | 2.212165 / 1.504120 (0.708045) | 1.894405 / 1.541195 (0.353210) | 1.922914 / 1.468490 (0.454424) | 0.762443 / 4.584777 (-3.822334) | 4.669214 / 3.745712 (0.923502) | 5.016066 / 5.269862 (-0.253796) | 3.128821 / 4.565676 (-1.436856) | 0.091541 / 0.424275 (-0.332734) | 0.007582 / 0.007607 (-0.000026) | 0.652753 / 0.226044 (0.426709) | 6.601375 / 2.268929 (4.332446) | 3.076948 / 55.444624 (-52.367677) | 2.250544 / 6.876477 (-4.625933) | 2.404059 / 2.142072 (0.261987) | 0.994917 / 4.805227 (-3.810311) | 0.200318 / 6.500664 (-6.300346) | 0.069354 / 0.075469 (-0.006115) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.482559 / 1.841788 (-0.359229) | 20.722092 / 8.074308 (12.647784) | 17.703217 / 10.191392 (7.511825) | 0.215370 / 0.680424 (-0.465053) | 0.028208 / 0.534201 (-0.505993) | 0.425992 / 0.579283 (-0.153291) | 0.492785 / 0.434364 (0.058421) | 0.474154 / 0.540337 (-0.066183) | 0.644599 / 1.386936 (-0.742337) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008372 / 0.011353 (-0.002981) | 0.004543 / 0.011008 (-0.006465) | 0.070564 / 0.038508 (0.032056) | 0.066855 / 0.023109 (0.043746) | 0.386724 / 0.275898 (0.110826) | 0.432184 / 0.323480 (0.108704) | 0.005250 / 0.007986 (-0.002736) | 0.003630 / 0.004328 (-0.000698) | 0.069310 / 0.004250 (0.065060) | 0.055759 / 0.037052 (0.018707) | 0.375789 / 0.258489 (0.117299) | 0.417335 / 0.293841 (0.123494) | 0.043424 / 0.128546 (-0.085122) | 0.013106 / 0.075646 (-0.062541) | 0.087836 / 0.419271 (-0.331436) | 0.057770 / 0.043533 (0.014237) | 0.396694 / 0.255139 (0.141555) | 0.439350 / 0.283200 (0.156150) | 0.031660 / 0.141683 (-0.110023) | 1.571339 / 1.452155 (0.119185) | 1.667169 / 1.492716 (0.174452) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.180534 / 0.018006 (0.162528) | 0.540027 / 0.000490 (0.539537) | 0.003573 / 0.000200 (0.003373) | 0.000141 / 0.000054 (0.000086) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031380 / 0.037411 (-0.006032) | 0.083762 / 0.014526 (0.069236) | 0.098166 / 0.176557 (-0.078390) | 0.160761 / 0.737135 (-0.576374) | 0.097683 / 0.296338 (-0.198656) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.568074 / 0.215209 (0.352865) | 5.660544 / 2.077655 (3.582889) | 2.416698 / 1.504120 (0.912578) | 2.177096 / 1.541195 (0.635901) | 2.206178 / 1.468490 (0.737688) | 0.844864 / 4.584777 (-3.739912) | 4.793636 / 3.745712 (1.047923) | 7.062387 / 5.269862 (1.792525) | 4.201228 / 4.565676 (-0.364449) | 0.091997 / 0.424275 (-0.332279) | 0.007881 / 0.007607 (0.000274) | 0.679466 / 0.226044 (0.453422) | 6.580268 / 2.268929 (4.311340) | 3.229907 / 55.444624 (-52.214717) | 2.524877 / 6.876477 (-4.351600) | 2.463796 / 2.142072 (0.321723) | 0.975627 / 4.805227 (-3.829600) | 0.186670 / 6.500664 (-6.313994) | 0.065307 / 0.075469 (-0.010163) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.501447 / 1.841788 (-0.340340) | 21.231037 / 8.074308 (13.156729) | 17.591671 / 10.191392 (7.400279) | 0.212745 / 0.680424 (-0.467679) | 0.026100 / 0.534201 (-0.508101) | 0.428391 / 0.579283 (-0.150892) | 0.535268 / 0.434364 (0.100904) | 0.506733 / 0.540337 (-0.033604) | 0.660832 / 1.386936 (-0.726104) |

![](https://cml.dev/watermark.png#962537d7ee9191438ef47a4185d0ba626b2ee949 "CML watermark")

Clarifies in the return statement of the docstring that the retrieval score is `IndexFlatL2` by default (see [PR](https://github.com/huggingface/transformers/issues/24739) and internal Slack [convo](https://huggingface.slack.com/archives/C01229B19EX/p1689105179711689)), and fixes the formatting because multiple return values are not supported.


Inconsistent "The features can't be aligned" error when combining map, multiprocessing, and variable length outputs

This scenario currently requires explicitly passing the target features (to avoid the error):
```python
import datasets

...

features = dataset.features
features["output"] = = [{"test": datasets.Value("int64")}]
test2 = dataset.map(lambda row, idx: test_func(row, idx), with_indices=True, num_proc=32, features=features)
```

### Describe the bug

I'm using a dataset with map and multiprocessing to run a function that returned a variable length list of outputs. This output list may be empty. Normally this is handled fine, but there is an edge case that crops up when using multiprocessing. In some cases, an empty list result ends up in a dataset shard consisting of a single item. This results in a `The features can't be aligned` error that is difficult to debug because it depends on the number of processes/shards used.

I've reproduced a minimal example below. My current workaround is to fill empty results with a dummy value that I filter after, but this was a weird error that took a while to track down.

### Steps to reproduce the bug

```python
import datasets

dataset = datasets.Dataset.from_list([{'idx':i} for i in range(60)])

def test_func(row, idx):
    if idx==58:
        return {'output': []}
    else:
        return {'output' : [{'test':1}, {'test':2}]}

# this works fine
test1 = dataset.map(lambda row, idx: test_func(row, idx), with_indices=True, num_proc=4)

# this fails
test2 = dataset.map(lambda row, idx: test_func(row, idx), with_indices=True, num_proc=32)
>ValueError: The features can't be aligned because the key output of features {'idx': Value(dtype='int64', id=None), 'output': Sequence(feature=Value(dtype='null', id=None), length=-1, id=None)} has unexpected type - Sequence(feature=Value(dtype='null', id=None), length=-1, id=None) (expected either [{'test': Value(dtype='int64', id=None)}] or Value("null").
```

The error occurs during the check

```python
_check_if_features_can_be_aligned([dset.features for dset in dsets])
```

When the multiprocessing splitting lines up just right with the empty return value, one of the `dset` in `dsets` will have a single item with an empty list value, causing the error.

### Expected behavior

Expected behavior is the result would be the same regardless of the `num_proc` value used.

### Environment info

Datasets version 2.11.0
Python 3.9.16