Use max number of data files to infer module

When inferring the module for datasets without script, set a maximum number of iterations over data files.

This PR fixes the issue of taking too long when hundred of data files present.

Please, feel free to agree on both numbers:
```
# Datasets without script
DATA_FILES_MAX_NUMBER = 10
ARCHIVED_DATA_FILES_MAX_NUMBER = 5
```

Fix #3404.


Fix module inference for archive with a directory

Fix module inference for an archive file that contains files within a directory.

Fix #3405.


ZIP format inference does not work when files located in a dir inside the archive

bug

Something isn't working

## Describe the bug
When a zipped file contains archived files within a directory, the function `infer_module_for_data_files_in_archives` does not work.

It only works for files located in the root directory of the ZIP file.

## Steps to reproduce the bug
```python
infer_module_for_data_files_in_archives(["path/to/zip/file.zip"], False)
```


Optimize ZIP format inference

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**
When hundreds of ZIP files are present in a dataset, format inference takes too long.

See: https://github.com/bigscience-workshop/data_tooling/issues/232#issuecomment-986685497

**Describe the solution you'd like**
Iterate over a maximum number of files.

CC: @lhoestq


Cannot import name 'maybe_sync'

bug

Something isn't working

## Describe the bug
Cannot seem to import datasets when running run_summarizer.py script on a VM set up on ovhcloud

## Steps to reproduce the bug
```python
from datasets import load_dataset
```

## Expected results
No error

## Actual results
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/lib/python3.7/site-packages/datasets/__init__.py", line 34, in <module>
    from .arrow_dataset import Dataset, concatenate_datasets
  File "/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 48, in <module>
    from .arrow_writer import ArrowWriter, OptimizedTypedSequence
  File "/opt/conda/lib/python3.7/site-packages/datasets/arrow_writer.py", line 27, in <module>
    from .features import (
  File "/opt/conda/lib/python3.7/site-packages/datasets/features/__init__.py", line 2, in <module>
    from .audio import Audio
  File "/opt/conda/lib/python3.7/site-packages/datasets/features/audio.py", line 8, in <module>
    from ..utils.streaming_download_manager import xopen
  File "/opt/conda/lib/python3.7/site-packages/datasets/utils/streaming_download_manager.py", line 16, in <module>
    from ..filesystems import COMPRESSION_FILESYSTEMS
  File "/opt/conda/lib/python3.7/site-packages/datasets/filesystems/__init__.py", line 13, in <module>
    from .s3filesystem import S3FileSystem  # noqa: F401
  File "/opt/conda/lib/python3.7/site-packages/datasets/filesystems/s3filesystem.py", line 1, in <module>
    import s3fs
  File "/opt/conda/lib/python3.7/site-packages/s3fs/__init__.py", line 1, in <module>
    from .core import S3FileSystem, S3File
  File "/opt/conda/lib/python3.7/site-packages/s3fs/core.py", line 11, in <module>
    from fsspec.asyn import AsyncFileSystem, sync, sync_wrapper, maybe_sync
ImportError: cannot import name 'maybe_sync' from 'fsspec.asyn' (/opt/conda/lib/python3.7/site-packages/fsspec/asyn.py)

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.16.0
- Platform: OVH Cloud Tesla V100 Machine
- Python version: 3.7.9
- PyArrow version: 6.0.1


More robust first elem check in encode/cast example

Fix #3306


Add Wikimedia pre-processed datasets

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** Add pre-processed data to:
  - *wikimedia/wikipedia*: https://huggingface.co/datasets/wikimedia/wikipedia
  - *wikimedia/wikisource*: https://huggingface.co/datasets/wikimedia/wikisource
- **Description:** Add pre-processed data to the Hub for all languages
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** This will be very useful for the NLP community, as the pre-processing has a high cost for lot of researchers (both in computation and in knowledge)

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

CC: @geohci, @yjernite


Improve Wikipedia loading script

Requesting to add a new dataset

Thanks! See https://public.paws.wmcloud.org/User:Isaac_(WMF)/HuggingFace%20Wikipedia%20Processing.ipynb for more implementation details / some data around the overhead induced by adding the extra preprocessing steps (stripping link prefixes and magic words)

As reported by @geohci, the "wikipedia" processing/loading script could be improved by some additional small suggested processing functions:
- _extract_content(filepath):
  - Replace .startswith("#redirect") with more structured approach: if elem.find(f"./{namespace}redirect") is None: continue
- _parse_and_clean_wikicode(raw_content, parser):
  - Remove rm_template from cleaning -- this is redundant with .strip_code() from mwparserformhell
  - Build a language-specific list of namespace prefixes to filter out per below get_namespace_prefixes
  - Optional: strip prefixes like categories -- e.g., Category:Towns in Tianjin becomes Towns in Tianjin
  - Optional: strip magic words


Add Wikisource dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *wikisource*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** Additional high quality textual data, besides Wikipedia.

Add loading script as "canonical" dataset (as it is the case for ""wikipedia").

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

CC: @geohci, @yjernite


Add URL field to Wikimedia dataset instances: wikipedia,...

Requesting to add a new dataset

As reported by @geohci, once we will host pre-processed data in the Hub, we should add the full URL to data instances (new field "url") in order to conform to proper attribution from license requirement. See, e.g.: https://fair-trec.github.io/docs/Fair_Ranking_2021_Participant_Instructions.pdf#subsection.3.2

This should be done for all pre-processed datasets under "wikimedia" org in the Hub: https://huggingface.co/wikimedia


add BNL processed newspapers

This pull request adds the BNL's [processed newspaper collections](https://data.bnl.lu/data/historical-newspapers/) as a dataset. This is partly done to support BigScience see: https://github.com/bigscience-workshop/data_tooling/issues/192.

The Datacard is more sparse than I would like but I plan to make a separate pull request to try and make this more complete at a later date.

I had to manually add the `dummy_data` but I believe I've done this correctly (the tests pass locally).


Install Audio dependencies to support audio decoding

dataset-viewer

Related to the dataset viewer on huggingface.co

## Dataset viewer issue for '*openslr*', '*projecte-aina/parlament_parla*'

**Link:** *https://huggingface.co/datasets/openslr*
**Link:** *https://huggingface.co/datasets/projecte-aina/parlament_parla*

Error:
```
Status code:   400
Exception:     ImportError
Message:       To support decoding audio files, please install 'librosa'.
```

Am I the one who added this dataset ? Yes-No
- openslr: No
- projecte-aina/parlament_parla: Yes


Fix formatting in IterableDataset.map docs

Fix formatting in the recently added `Map` section of the streaming docs.


Preserve all feature types when saving a dataset on the Hub with `push_to_hub`

bug

Something isn't working

Currently, if one of the dataset features is of type `ClassLabel`, saving the dataset with `push_to_hub` and reloading the dataset with `load_dataset` will return the feature of type `Value`. To fix this, we should do something similar to `save_to_disk` (which correctly preserves the types) and not only push the parquet files in `push_to_hub`, but also the dataset `info` (stored in a JSON file).


Common Voice Belarusian Dataset

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *Common Voice Belarusian Dataset*
- **Description:** *[commonvoice.mozilla.org/be](https://commonvoice.mozilla.org/be)*
- **Data:** *[commonvoice.mozilla.org/be/datasets](https://commonvoice.mozilla.org/be/datasets)*
- **Motivation:** *It has more than 7GB of data, so it will be great to have it in this package so anyone can try to train something for Belarusian language.*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


This issue was fixed by me calling `all_datasets.push_to_hub("hackernews_hiring_posts")`.

The previous problems were from calling `all_datasets.save_to_disk` and then pushing with `my_repo.git_add` and `my_repo.push_to_hub`.

## Dataset viewer issue for `dansbecker/hackernews_hiring_posts`

**Link:** https://huggingface.co/datasets/dansbecker/hackernews_hiring_posts

*short description of the issue*

Dataset preview not showing for uploaded DatasetDict. See https://discuss.huggingface.co/t/dataset-preview-not-showing-for-uploaded-datasetdict/12603

Am I the one who added this dataset ?

No -> @dansbecker


method to select columns

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**

* There is currently no way to select some columns of a dataset. In pandas, one can use `df[['col1', 'col2']]` to select columns, but in `datasets`, it results in error.

**Describe the solution you'd like**

* A new method that can be used to create a new dataset with only a list of specified columns.

**Describe alternatives you've considered**

`.remove_columns(self, columns: Union[str, List[str]], inverse: bool = False)`
Or
`.select(self,  indices: Iterable = None, columns: List[str] = None)`


Loading dataset throws "KeyError: 'Field "builder_name" does not exist in table schema'"

bug

Something isn't working

## Describe the bug
I have prepared dataset to datasets and now I am trying to load it back Finnish-NLP/voxpopuli_fi
I get "KeyError: 'Field "builder_name" does not exist in table schema'"

My dataset folder and files should be like @patrickvonplaten  has here https://huggingface.co/datasets/flax-community/german-common-voice-processed

How my voxpopuli dataset looks like:
![image](https://user-images.githubusercontent.com/25264037/144895598-b7d9ae91-b04a-4046-9f06-b71ff0824d13.png)

Part of the processing (path column is the absolute path to audio files)
```
def add_audio_column(example):
    example['audio'] = example['path']
    return example

voxpopuli = voxpopuli.map(add_audio_column)
voxpopuli.cast_column("audio", Audio())
voxpopuli["audio"] <-- to my knowledge this does load the local files and prepares those arrays
voxpopuli = voxpopuli.cast_column("audio", Audio(sampling_rate=16_000)) resampling 16kHz
```

I have then saved it to disk_
`voxpopuli.save_to_disk('/asr_disk/datasets_processed_new/voxpopuli')`

and made folder structure same as @patrickvonplaten
I also get same error while trying to load_dataset from his repo:
![image](https://user-images.githubusercontent.com/25264037/144895872-e9b8f326-cf2b-46cf-9417-606a0ce14077.png)

## Steps to reproduce the bug
```python
dataset = load_dataset("Finnish-NLP/voxpopuli_fi")
```

## Expected results
Dataset is loaded correctly and looks like in the first picture

## Actual results
Loading throws keyError:
KeyError: 'Field "builder_name" does not exist in table schema'

Resources I have been trying to follow:
https://huggingface.co/docs/datasets/audio_process.html
https://huggingface.co/docs/datasets/share_dataset.html

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.16.2.dev0
- Platform: Ubuntu 20.04.2 LTS
- Python version: 3.8.12
- PyArrow version: 6.0.1


Add EDGAR

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** EDGAR Database
- **Description:** https://www.sec.gov/edgar/about EDGAR, the Electronic Data Gathering, Analysis, and Retrieval system, is the primary system for companies and others submitting documents under the Securities Act of 1933, the Securities Exchange Act of 1934, the Trust Indenture Act of 1939, and the Investment Company Act of 1940. Containing millions of company and individual filings, EDGAR benefits investors, corporations, and the U.S. economy overall by increasing the efficiency, transparency, and fairness of the securities markets. The system processes about 3,000 filings per day, serves up 3,000 terabytes of data to the public annually, and accommodates 40,000 new filers per year on average. EDGAR® and EDGARLink® are registered trademarks of the SEC.
- **Data:** https://www.sec.gov/os/accessing-edgar-data
- **Motivation:** Enabling and improving FSI (Financial Services Industry) datasets to increase ease of use

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Fix flaky test of the temporary directory used by load_from_disk

CI failed because of a server error - merging

The test is flaky, here is an example of random CI failure:
https://github.com/huggingface/datasets/commit/73ed6615b4b3eb74d5311684f7b9e05cdb76c989

I fixed that by not checking the content of the random part of the temporary directory name


Create Language Modeling task

Create Language Modeling task to be able to specify the input "text" column in a dataset.

This can be useful for datasets which are not exclusively used for language modeling and have more than one column:
- for text classification datasets (with columns "review" and "rating", for example), the Language Modeling task can be used to specify the "text" column ("review" in this case).

TODO:
- [ ] Add the LanguageModeling task to all dataset scripts which can be used for language modeling


Fix typos in dataset cards

This PR:
- Fix typos in dataset cards
- Fix Papers With Code ID for:
  - Bilingual Corpus of Arabic-English Parallel Tweets
  - Tweets Hate Speech Detection
- Add pretty name tags


None batched `with_transform`, `set_transform`

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**

A `torch.utils.data.Dataset.__getitem__` operates on a single example.
But 🤗 `Datasets.with_transform` doesn't seem to allow non-batched transform.

**Describe the solution you'd like**

Have a `batched=True` argument in `Datasets.with_transform`

**Describe alternatives you've considered**

* Convert a non-batched transform function to batched one myself.
* Wrap a 🤗 Dataset with torch Dataset, and add a `__getitem__`. 🙄
* Have `lazy=False` in `Dataset.map`, and returns a `LazyDataset` if `lazy=True`. This way the same `map` interface can be used, and existing code can be updated with one argument change.


Adding mMARCO dataset

We are adding mMARCO dataset to HuggingFace datasets repo.
This way, all the languages covered in the translation are available in a easy way.


add Georgian data in cc100.

update cc100 dataset to support loading Georgian (ka) data which is originally available in CC100 dataset source.
All tests are passed.
Dummy data generated.
metadata generated.


#3337 Add typing overloads to Dataset.__getitem__ for mypy

Locally the `make quality` passes with the same dependencies. I would suggest upgrading flake8. (I can take care of it in another PR)
cc @lhoestq

Add typing overloads to Dataset.__getitem__ for mypy

Fixes #3337

**Iterable**
Iterable from `collections` cannot have a type, so you can't do `Iterable[int]` for example. `typing` has a Generic version that builds upon the one from `collections`.

**Flake8**
I had to add `# noqa: F811`, this is a bug from Flake8.

datasets uses flake8==3.7.9 which released in October 2019 if I update flake8 (4.0.1), I no longer get these errors, but I did not want to make the update without your approval. (It also triggers other errors like no args in f-strings.)


Unable to load audio_features from common_voice dataset

bug

Something isn't working

Hi ! Feel free to access `batch["audio"]["array"]` and `batch["audio"]["sampling_rate"]` instead

`datasets` 1.16 introduced some changes in `common_voice` and now the `path` field is no longer a path to a local file (but rather the path to the file in the archive it's extracted from)

## Describe the bug
I am not able to load audio features from common_voice dataset

## Steps to reproduce the bug

```
from datasets import load_dataset
import torchaudio

test_dataset = load_dataset("common_voice", "hi", split="test[:2%]")
resampler = torchaudio.transforms.Resample(48_000, 16_000)

def speech_file_to_array_fn(batch):
      speech_array, sampling_rate = torchaudio.load(batch["path"])
      batch["speech"] = resampler(speech_array).squeeze().numpy()
      return batch
test_dataset = test_dataset.map(speech_file_to_array_fn)
```
## Expected results

This piece of code should return test_dataset after loading audio features.

## Actual results

Reusing dataset common_voice (/home/jovyan/.cache/huggingface/datasets/common_voice/hi/6.1.0/b879a355caa529b11f2249400b61cadd0d9433f334d5c60f8c7216ccedfecfe1)
/opt/conda/lib/python3.7/site-packages/transformers/configuration_utils.py:341: UserWarning: Passing `gradient_checkpointing` to a config initialization is deprecated and will be removed in v5 Transformers. Using `model.gradient_checkpointing_enable()` instead, or if you are using the `Trainer` API, pass `gradient_checkpointing=True` in your `TrainingArguments`.
  "Passing `gradient_checkpointing` to a config initialization is deprecated and will be removed in v5 "
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
  0%|                                                                                | 0/3 [00:00<?, ?ex/s]formats: can't open input file `common_voice_hi_23795358.mp3': No such file or directory
  0%|                                                                                | 0/3 [00:00<?, ?ex/s]
Traceback (most recent call last):
  File "demo_file.py", line 23, in <module>
    test_dataset = test_dataset.map(speech_file_to_array_fn)
  File "/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 2036, in map
    desc=desc,
  File "/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 518, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 485, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/datasets/fingerprint.py", line 411, in wrapper
    out = func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 2368, in _map_single
    example = apply_function_on_filtered_inputs(example, i, offset=offset)
  File "/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 2277, in apply_function_on_filtered_inputs
    processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
  File "/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 1978, in decorated
    result = f(decorated_item, *args, **kwargs)
  File "demo_file.py", line 19, in speech_file_to_array_fn
    speech_array, sampling_rate = torchaudio.load(batch["path"])
  File "/opt/conda/lib/python3.7/site-packages/torchaudio/backend/sox_io_backend.py", line 154, in load
    filepath, frame_offset, num_frames, normalize, channels_first, format)
RuntimeError: Error loading audio file: failed to open file common_voice_hi_23795358.mp3

## Environment info

- `datasets` version: 1.16.1
- Platform: Linux-4.14.243 with-debian-bullseye-sid
- Python version: 3.7.9
- PyArrow version: 6.0.1


[Quick poll] Give your opinion on the future of the Hugging Face Open Source ecosystem!

Thanks to all of you, `datasets` will pass 11.5k stars :star2: this week!

If you have a couple of minutes and want to participate in shaping the future of the ecosystem, please share your thoughts:

[**hf.co/oss-survey**](https://hf.co/oss-survey)

(please reply in the above feedback form rather than to this thread)

Thank you all on behalf of the HuggingFace team! 🤗


iter_archive on zipfiles with better compression type check

Hello @lhoestq, thank you for your answer.

I don't use pytest a lot so I think I might need some help on it :) but I tried some tests for `streaming_download_manager.py` only. I don't know how to test `download_manager.py` since we need to use local files.

# Comments :
* In **download_manager.py** I removed some unnecessary imports after the simplification of `_get_extraction_protocol_local`.
* In **streaming_download_manager** I moved the raised Error as suggested.

### I also started some tests on `StreamingDownloadManager()` :
* Used an existing zipfile url and added a new one that has a folder and many files :
```python
TEST_GG_DRIVE_ZIPPED_URL = "https://drive.google.com/uc?export=download&id=1k92sUfpHxKq8PXWRr7Y5aNHXwOCNUmqh"
TEST_GG_DRIVE2_ZIPPED_URL = "https://drive.google.com/uc?export=download&id=1X4jyUBBbShyCRfD-vCO1ZvfqFXP3NEeU"
```
* **For now is being tested :**
     * Return type of the function  : should be tuple
     * Files names
     * Files content
 * Added an `xfail` test for the gzip file, because I get a `zipfile.BadZipFile exception`.

 * And lastly, changed the test for `_get_extraction_protocol_throws` since it was moved to `_extract` :
 ```diff
@pytest.mark.xfail(raises=NotImplementedError)
def test_streaming_dl_manager_get_extraction_protocol_throws(urlpath):
-    _get_extraction_protocol(urlpath)

@pytest.mark.xfail(raises=NotImplementedError)
def test_streaming_dl_manager_get_extraction_protocol_throws(urlpath):
+    StreamingDownloadManager()._extract(urlpath)
```

Hello @lhoestq , thank you for your detailed answer on previous PR !
I made this new PR because I misused git on the previous one #3347.
Related issue #3272.

# Comments  :

* For extension check I used the `_get_extraction_protocol` function in **download_manager.py** with a slight change and called it `_get_extraction_protocol_local`:

**I removed this part :**
 ```python
    elif path.endswith(".tar.gz") or path.endswith(".tgz"):
        raise NotImplementedError(
            f"Extraction protocol for TAR archives like '{urlpath}' is not implemented in streaming mode. Please use `dl_manager.iter_archive` instead."
        )
```
**And also changed :**
```diff
- extension = path.split(".")[-1]
+ extension = "tar" if path.endswith(".tar.gz") else path.split(".")[-1]
```
The reason for this is a compression like **.tar.gz** will be considered a **.gz** which is handled with **zipfile**, though **tar.gz** can only be opened using **tarfile**.

Please tell me if there's anything to change.

# Tasks :
- [x] download_manager.py
- [x] streaming_download_manager.py


Add The Pile subsets

Add The Pile subsets:
- pubmed
- ubuntu_irc
- europarl
- hacker_news
- nih_exporter

Close bigscience-workshop/data_tooling#301.

CC: @StellaAthena


COCO 🥥 on the 🤗 Hub?

TO-DO:
- [x] Add unlabeled 2017 splits, train and validation splits of 2015
- [ ] Add Class Labels as list instead

This is a draft PR since I ran into few small problems.
I referred to this TFDS code: https://github.com/tensorflow/datasets/blob/2538a08c184d53b37bfcf52cc21dd382572a88f4/tensorflow_datasets/object_detection/coco.py
cc: @mariosasko


Update clue benchmark

The CI error is due to missing tags in the CLUE dataset card - merging !

Fix #3374


Support streaming zipped dataset repo by passing only repo name

I just tested and I think this only opens one file ? If there are several files in the ZIP, only the first one is opened. To open several files from a ZIP, one has to call `open` several times.

What about updating the CSV loader to make it `download_and_extract` zip files, and open each extracted file ?

I have implemented the  glob of ZIP files in the packaged modules:
- csv
- json
- text

Fix #3373.


NonMatchingChecksumError for the CLUE:cluewsc2020, chid, c3 and tnews

Seems like the issue still exists,:
`Downloading and preparing dataset clue/chid (download: 127.15 MiB, generated: 259.71 MiB, post-processed: Unknown size, total: 386.86 MiB) to /mnt/cache/tanhaochen/.cache/huggingface/datasets/clue/chid/1.0.0/e55b490cb7809dcd8db31b9a87119f2e2ec87cdc060da8a9ac070b070ca3e379...
Traceback (most recent call last):
  File "/mnt/cache/tanhaochen/PromptCLUE/test_datasets.py", line 3, in <module>
    cluewsc2020 = datasets.load_dataset("clue","chid")
  File "/mnt/cache/tanhaochen/dependencies/datasets/src/datasets/load.py", line 1667, in load_dataset
    builder_instance.download_and_prepare(
  File "/mnt/cache/tanhaochen/dependencies/datasets/src/datasets/builder.py", line 593, in download_and_prepare
    self._download_and_prepare(
  File "/mnt/cache/tanhaochen/dependencies/datasets/src/datasets/builder.py", line 663, in _download_and_prepare
    verify_checksums(
  File "/mnt/cache/tanhaochen/dependencies/datasets/src/datasets/utils/info_utils.py", line 40, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://storage.googleapis.com/cluebenchmark/tasks/chid_public.zip']
`

Hi,

the fix hasn't been merged yet (it should be merged early next week).

Hi, it seems like there are updates in cluewsc2020, chid, c3 and tnews, since i could not load them due to the checksum error.


Support streaming zipped CSV dataset repo by passing only repo name

enhancement

New feature or request

Given a community 🤗 dataset repository containing only a zipped CSV file (only raw data, no loading script), I would like to load it in streaming mode without passing `data_files`:
```
ds_name = "bigscience-catalogue-data/vietnamese_poetry_from_fsoft_ai_lab"
ds = load_dataset(ds_name, split="train", streaming=True, use_auth_token=True)
item = next(iter(ds))
```

Currently, it gives a `FileNotFoundError` because there is no glob (no "\*" after "zip://": "zip://*") in the passed URL:
```
'zip://::https://huggingface.co/datasets/bigscience-catalogue-data/vietnamese_poetry_from_fsoft_ai_lab/resolve/e5d45f1bd9a8a798cc14f0a45ebc1ce91907c792/poems_dataset.zip'
```


[SEO improvement] Add Dataset Metadata to make datasets indexable

New feature or request

Some people who host datasets on github seem to include a table of metadata at the end of their README.md to make the dataset indexable by [Google Dataset Search](https://datasetsearch.research.google.com/) (See [here](https://github.com/google-research/google-research/tree/master/goemotions#dataset-metadata) and [here](https://github.com/cvdfoundation/google-landmark#dataset-metadata)). This could be a useful addition to canonical datasets; perhaps even community datasets.

I'll include a screenshot (as opposed to markdown) as an example so as not to have a github issue indexed as a dataset:

> ![image](https://user-images.githubusercontent.com/3664563/144496173-953428cf-633a-4571-b75b-f099c6b2ed65.png)

**_PS: It might very well be the case that this is already covered by some other markdown magic I'm not aware of._**


New: Americas NLI dataset

This PR adds the [Americas NLI](https://arxiv.org/abs/2104.08726) dataset, extension of XNLI to 10 low-resource indigenous languages spoken in the Americas: Ashaninka, Aymara, Bribri, Guarani, Nahuatl, Otomi, Quechua, Raramuri, Shipibo-Konibo, and Wixarika.

One odd thing (not sure) is that I had to set

`datasets-cli dummy_data ./datasets/americas_nli/ --auto_generate --n_lines 7500`

`n_lines` very large to successfully generate the dummy files for all the subsets. Happy to get some guidance here.

Otherwise, I hope everything is in order :)

e: missed a step, onto fixing the tests
e2: there you go -- hope it's ok to have added more languages with their ISO codes to `languages.json`, need those tests to pass :laughing:


Document a training loop for streaming dataset

I added some docs about streaming dataset. In particular I added two subsections:
- one on how to use `map` for preprocessing
- one on how to use a streaming dataset in a pytorch training loop

cc @patrickvonplaten @stevhliu if you have some comments
cc @Rocketknight1 later we can add the one for TF and I might need your help ^^'


[Audio] Allow resampling for audio datasets in streaming mode

New feature or request

Many audio datasets like Common Voice always need to be resampled. This can very easily be done in non-streaming mode as follows:

```python
from datasets import load_dataset

ds = load_dataset("common_voice", "ab", split="test")

ds = ds.cast_column("audio", Audio(sampling_rate=16_000))
```

However in streaming mode it fails currently:

```python
from datasets import load_dataset

ds = load_dataset("common_voice", "ab", split="test", streaming=True)

ds = ds.cast_column("audio", Audio(sampling_rate=16_000))
```

with the following error:

```
AttributeError: 'IterableDataset' object has no attribute 'cast_column'
```

It would be great if we could add such a feature (I'm not 100% sure though how complex this would be)


Fix dict source_datasets tagset validator

Currently, the `source_datasets` tag validation does not support passing a dict with configuration keys.

This PR:
- Extends `tagset_validator` to support regex tags
- Uses `tagset_validator` to validate dict `source_datasets`


Fix typo in other-structured-to-text task tag

Fix typo in task tag:
- `other-stuctured-to-text` (before)
- `other-structured-to-text` (now)


Add multimodal datasets

dataset request

Requesting to add a new dataset

Epic issue to track the addition of multimodal datasets:
- [ ] #2526
- [ ] #1842
- [ ] #1810

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

@VictorSanh feel free to add and sort by priority any interesting dataset. I have added the multimodal dataset requests which were already present as issues.


Add task tags for multimodal datasets

enhancement

New feature or request

## **Is your feature request related to a problem? Please describe.**

Currently, task tags are either exclusively related to text or speech processing:
- https://github.com/huggingface/datasets/blob/master/src/datasets/utils/resources/tasks.json

## **Describe the solution you'd like**

We should also add tasks related to:
- multimodality
- image
- video

CC: @VictorSanh @lewtun @lhoestq @merveenoyan @SBrandeis


Use the Audio feature in the AutomaticSpeechRecognition template

This updates the ASR template and all supported datasets to use the `Audio` feature


Update URL of Jeopardy! dataset

Updates the URL of the Jeopardy! dataset.

Fix #3361


Adapt image datasets

This PR can be merged after #3163 is merged (this PR is pretty big because I was working on the forked branch).

@lhoestq @albertvillanova Could you please take a look at the changes in `src/datasets/utils/streaming_download_manager.py`? These changes were required to support streaming of the `cats_vs_dogs` and the `beans` datasets.

The CI failures are due to the missing fields in the README files.

This PR:
* adapts the ImageClassification template to use the new Image feature
* adapts the following datasets to use the new Image feature:
  * beans (+ fixes streaming)
  * cast_vs_dogs (+ fixes streaming)
  * cifar10
  * cifar100
  * fashion_mnist
  * mnist
  * head_qa

cc @nateraw


Jeopardy _URL access denied

bug

Something isn't working

## Describe the bug
http://skeeto.s3.amazonaws.com/share/JEOPARDY_QUESTIONS1.json.gz returns Access Denied now.

However, https://drive.google.com/file/d/0BwT5wj_P7BKXb2hfM3d2RHU1ckE/view?usp=sharing from the original Reddit post https://www.reddit.com/r/datasets/comments/1uyd0t/200000_jeopardy_questions_in_a_json_file/ may work.

## Steps to reproduce the bug
```shell
> python
Python 3.7.12 (default, Sep  5 2021, 08:34:29)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
```
```python
>>> from datasets import load_dataset
>>> load_dataset("jeopardy")
```

## Expected results
The download completes.

## Actual results
```shell
Downloading: 4.18kB [00:00, 1.60MB/s]
Downloading: 2.03kB [00:00, 1.04MB/s]
Using custom data configuration default
Downloading and preparing dataset jeopardy/default (download: 12.13 MiB, generated: 34.46 MiB, post-processed: Unknown size, total: 46.59 MiB) to /Users/mike/.cache/huggingface/datasets/jeopardy/default/0.1.0/25ee3e4a73755e637b8810f6493fd36e4523dea3ca8a540529d0a6e24c7f9810...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/load.py", line 1632, in load_dataset
    use_auth_token=use_auth_token,
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/builder.py", line 608, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/builder.py", line 675, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/Users/mike/.cache/huggingface/modules/datasets_modules/datasets/jeopardy/25ee3e4a73755e637b8810f6493fd36e4523dea3ca8a540529d0a6e24c7f9810/jeopardy.py", line 72, in _split_generators
    filepath = dl_manager.download_and_extract(_DATA_URL)
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 284, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 197, in download
    download_func, url_or_urls, map_tuple=True, num_proc=download_config.num_proc, disable_tqdm=False
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/utils/py_utils.py", line 197, in map_nested
    return function(data_struct)
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 217, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 305, in cached_path
    use_auth_token=download_config.use_auth_token,
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 594, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach http://skeeto.s3.amazonaws.com/share/JEOPARDY_QUESTIONS1.json.gz
```

---

```shell
> curl http://skeeto.s3.amazonaws.com/share/JEOPARDY_QUESTIONS1.json.gz
```
```xml
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>70Y9R36XNPEQXMGV</RequestId><HostId>G6F5AK4qo7JdaEdKGMtS0P6gdLPeFOdEfSEfvTOZEfk9km0/jAfp08QLfKSTFFj1oWIKoAoBehM=</HostId></Error>
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.14.0
- Platform: macOS Catalina 10.15.7
- Python version: 3.7.12
- PyArrow version: 6.0.1


Add The Pile USPTO subset

Add:
- USPTO subset of The Pile: "uspto" config

Close bigscience-workshop/data_tooling#297.

CC: @StellaAthena


Add The Pile Free Law subset

@albertvillanova Is there a specific reason you’re adding the Pile under “the” instead of under “pile”? That does not appear to be consistent with other datasets.

Hi @StellaAthena,

I asked myself the same question, but at the end I decided to be consistent with previously added Pile subsets:
- #2817

I guess the reason is to stress that the definite article is always used before the name of the dataset (your site says: "The Pile. An 800GB Dataset of Diverse Text for Language Modeling"). Other datasets are not usually preceded by the definite article, like "the SQuAD" or "the GLUE" or "the Common Voice"...

CC: @lhoestq

> I guess the reason is to stress that the definite article is always used before the name of the dataset (your site says: "The Pile. An 800GB Dataset of Diverse Text for Language Modeling").

Yes that's because of this that it starts with "the"

Add:
- Free Law subset of The Pile: "free_law" config

Close bigscience-workshop/data_tooling#75.

CC: @StellaAthena


add new field, and get errors

Hi,

could you please post this question on our [Forum](https://discuss.huggingface.co/) as we keep issues for bugs and feature requests?

> Hi,
>
> could you please post this question on our [Forum](https://discuss.huggingface.co/) as we keep issues for bugs and feature requests?

ok.

after adding new field **tokenized_examples["example_id"]**, and get errors below,
I think it is due to changing data to tensor, and **tokenized_examples["example_id"]** is string list
**all fields**
```
***************** train_dataset 1: Dataset({
    features: ['attention_mask', 'end_positions', 'example_id', 'input_ids', 'start_positions', 'token_type_ids'],
    num_rows: 87714
})
```

**Errors**
```
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 705, in convert_to_tensors
    tensor = as_tensor(value)
ValueError: too many dimensions 'str'
```


Update README.md

After having worked a bit with the dataset.
As far as I know, it is solely in English (en-US). There are only a few mails in Spanish, French or German (less than a dozen I would estimate).


to_tf_dataset() refactor

Also, please don't merge yet - I need to make sure all the code samples and notebooks have a collate_fn specified, since we're removing the ability for this method to work without one!

Hi @lhoestq @mariosasko, the other PRs this was depending on in Transformers and huggingface/notebooks are now merged, so this is ready to go. Do you want to take one more look at it, or are you happy at this point?

The documentation for the method is fine, it doesn't need to be changed, but the tutorial notebook definitely looks a little out of date. Let me see what I can do!

@lhoestq I rewrote the last bit of the notebook - let me know what you think!

This is the promised cleanup to `to_tf_dataset()` now that the course is out of the way! The main changes are:

- A collator is always required (there was way too much hackiness making things like labels work without it)
- Lots of cleanup and a lot of code moved to `_get_output_signature`
- Should now handle it gracefully when the data collator adds unexpected columns


Extend support for streaming datasets that use pd.read_excel

This PR fixes error:
```
ValueError: Cannot seek streaming HTTP file
```

CC: @severo


Remove duplicate name from dataset cards

Remove duplicate name from dataset card for:
- ajgt_twitter_ar
- emotone_ar


add one field  "example_id", but I can't see it in the "comput_loss" function

Hi ! Your function looks fine, I used to map `squad` locally and it indeed added the `example_id` field correctly.

However I think that in the `compute_loss` method only a subset of the fields are available: the model inputs. Since `example_id` is not a model input (it's not passed as a parameter to the model), the data loader doesn't need to return it by default.

However you can disable this behavior by setting `remove_unused_columns` to `False` to your training arguments. In this case in `compute_loss` you will get the full item with all the fields.

Note that since the model doesn't take `example_id` as input, you will have to remove it from the inputs when `model(**inputs)` is called

Hi, I have set **args.remove_unused_columns=False** and  **training_args.remove_unused_columns=False**, but the field doesn't been contained yet.
```
def main():
    argp = HfArgumentParser(TrainingArguments)
    # The HfArgumentParser object collects command-line arguments into an object (and provides default values for unspecified arguments).
    # In particular, TrainingArguments has several keys that you'll need/want to specify (when you call run.py from the command line):
    # --do_train
    #     When included, this argument tells the script to train a model.
    #     See docstrings for "--task" and "--dataset" for how the training dataset is selected.
    # --do_eval
    #     When included, this argument tells the script to evaluate the trained/loaded model on the validation split of the selected dataset.
    # --per_device_train_batch_size <int, default=8>
    #     This is the training batch size.
    #     If you're running on GPU, you should try to make this as large as you can without getting CUDA out-of-memory errors.
    #     For reference, with --max_length=128 and the default ELECTRA-small model, a batch size of 32 should fit in 4gb of GPU memory.
    # --num_train_epochs <float, default=3.0>
    #     How many passes to do through the training data.
    # --output_dir <path>
    #     Where to put the trained model checkpoint(s) and any eval predictions.
    #     *This argument is required*.

    argp.add_argument('--model', type=str,
                      default='google/electra-small-discriminator',
                      help="""This argument specifies the base model to fine-tune.
        This should either be a HuggingFace model ID (see https://huggingface.co/models)
        or a path to a saved model checkpoint (a folder containing config.json and pytorch_model.bin).""")
    argp.add_argument('--task', type=str, choices=['nli', 'qa'], required=True,
                      help="""This argument specifies which task to train/evaluate on.
        Pass "nli" for natural language inference or "qa" for question answering.
        By default, "nli" will use the SNLI dataset, and "qa" will use the SQuAD dataset.""")
    argp.add_argument('--dataset', type=str, default=None,
                      help="""This argument overrides the default dataset used for the specified task.""")
    argp.add_argument('--max_length', type=int, default=128,
                      help="""This argument limits the maximum sequence length used during training/evaluation.
        Shorter sequence lengths need less memory and computation time, but some examples may end up getting truncated.""")
    argp.add_argument('--max_train_samples', type=int, default=None,
                      help='Limit the number of examples to train on.')
    argp.add_argument('--max_eval_samples', type=int, default=None,
                      help='Limit the number of examples to evaluate on.')

    argp.remove_unused_columns = False
    training_args, args = argp.parse_args_into_dataclasses()
    args.remove_unused_columns=False
    training_args.remove_unused_columns=False
```

```
**************** train_dataset: Dataset({
    features: ['id', 'title', 'context', 'question', 'answers'],
    num_rows: 87599
})

**************** train_dataset_featurized: Dataset({
    features: ['attention_mask', 'end_positions', 'input_ids', 'start_positions', 'token_type_ids'],
    num_rows: 87714
})
```

Hi, I print the value, all are set to False, but don't work.
```
********************* training_args: TrainingArguments(
_n_gpu=1,
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
dataloader_drop_last=False,
dataloader_num_workers=0,
dataloader_pin_memory=True,
ddp_find_unused_parameters=None,
debug=[],
deepspeed=None,
disable_tqdm=False,
do_eval=False,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_steps=None,
evaluation_strategy=IntervalStrategy.NO,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
gradient_accumulation_steps=1,
greater_is_better=None,
group_by_length=False,
ignore_data_skip=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=5e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=-1,
log_level=-1,
log_level_replica=-1,
log_on_each_node=True,
logging_dir=./re_trained_model/runs/Dec01_14-15-08_399b9290604c,
logging_first_step=False,
logging_steps=500,
logging_strategy=IntervalStrategy.STEPS,
lr_scheduler_type=SchedulerType.LINEAR,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
no_cuda=False,
num_train_epochs=3.0,
output_dir=./re_trained_model,
overwrite_output_dir=False,
past_index=-1,
per_device_eval_batch_size=8,
per_device_train_batch_size=8,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=re_trained_model,
push_to_hub_organization=None,
push_to_hub_token=None,
remove_unused_columns=False,
report_to=['tensorboard'],
resume_from_checkpoint=None,
run_name=./re_trained_model,
save_on_each_node=False,
save_steps=500,
save_strategy=IntervalStrategy.STEPS,
save_total_limit=None,
seed=42,
sharded_ddp=[],
skip_memory_metrics=True,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_legacy_prediction_loop=False,
warmup_ratio=0.0,
warmup_steps=0,
weight_decay=0.0,
)
```
```
********************* args: Namespace(dataset='squad', max_eval_samples=None, max_length=128, max_train_samples=None, model='google/electra-small-discriminator', remove_unused_columns=False, task='qa')
2021-12-01 14:15:10,048 - WARNING - datasets.builder - Reusing dataset squad (/root/.cache/huggingface/datasets/squad/plain_text/1.0.0/d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453)
Some weights of the model checkpoint at google/electra-small-discriminator were not used when initializing ElectraForQuestionAnswering: ['discriminator_predictions.dense_prediction.weight', 'discriminator_predictions.dense_prediction.bias', 'discriminator_predictions.dense.weight', 'discriminator_predictions.dense.bias']
- This IS expected if you are initializing ElectraForQuestionAnswering from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing ElectraForQuestionAnswering from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of ElectraForQuestionAnswering were not initialized from the model checkpoint at google/electra-small-discriminator and are newly initialized: ['qa_outputs.bias', 'qa_outputs.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Preprocessing data... (this takes a little bit, should only happen once per dataset)
```

Hmmm, it might be because the default data collator removes all the fields with `string` type:

https://github.com/huggingface/transformers/blob/4c0dd199c8305903564c2edeae23d294edd4b321/src/transformers/data/data_collator.py#L107-L112

I guess you also need a custom data collator that doesn't remove them.

can you give a tutorial about how to do this?

I overwrite **get_train_dataloader**, and remove **_remove_unused_columns**, but it doesn't work.

```
    def get_train_dataloader(self) -> DataLoader:
        """
        Returns the training :class:`~torch.utils.data.DataLoader`.

        Will use no sampler if :obj:`self.train_dataset` does not implement :obj:`__len__`, a random sampler (adapted
        to distributed training if necessary) otherwise.

        Subclass and override this method if you want to inject some custom behavior.
        """
        if self.train_dataset is None:
            raise ValueError("Trainer: training requires a train_dataset.")

        train_dataset = self.train_dataset
        # if is_datasets_available() and isinstance(train_dataset, datasets.Dataset):
        #     train_dataset = self._remove_unused_columns(train_dataset, description="training")

        if isinstance(train_dataset, torch.utils.data.IterableDataset):
            if self.args.world_size > 1:
                train_dataset = IterableDatasetShard(
                    train_dataset,
                    batch_size=self.args.train_batch_size,
                    drop_last=self.args.dataloader_drop_last,
                    num_processes=self.args.world_size,
                    process_index=self.args.process_index,
                )

            return DataLoader(
                train_dataset,
                batch_size=self.args.train_batch_size,
                collate_fn=self.data_collator,
                num_workers=self.args.dataloader_num_workers,
                pin_memory=self.args.dataloader_pin_memory,
            )

        train_sampler = self._get_train_sampler()

        return DataLoader(
            train_dataset,
            batch_size=self.args.train_batch_size,
            sampler=train_sampler,
            collate_fn=self.data_collator,
            drop_last=self.args.dataloader_drop_last,
            num_workers=self.args.dataloader_num_workers,
            pin_memory=self.args.dataloader_pin_memory,
        )
```

Hi, it works now, thank you.
1. **args.remove_unused_columns=False** and **training_args.remove_unused_columns=False**
2. overwrite **get_train_dataloader**, and remove **_remove_unused_columns**
3. add new fields, and can be got in **inputs**.

Hi,  I add one field **example_id**, but I can't see it in the **comput_loss** function, how can I do this? below is the information of inputs

```
*********************** inputs: {'attention_mask': tensor([[1, 1, 1,  ..., 0, 0, 0],
        [1, 1, 1,  ..., 0, 0, 0],
        [1, 1, 1,  ..., 0, 0, 0],
        ...,
        [1, 1, 1,  ..., 0, 0, 0],
        [1, 1, 1,  ..., 0, 0, 0],
        [1, 1, 1,  ..., 0, 0, 0]], device='cuda:0'), 'end_positions': tensor([ 25,  97,  93,  44,  25, 112, 109, 134], device='cuda:0'), 'input_ids': tensor([[ 101, 2054, 2390,  ...,    0,    0,    0],
        [ 101, 2054, 2515,  ...,    0,    0,    0],
        [ 101, 2054, 2106,  ...,    0,    0,    0],
        ...,
        [ 101, 2339, 2001,  ...,    0,    0,    0],
        [ 101, 2054, 2515,  ...,    0,    0,    0],
        [ 101, 2054, 2003,  ...,    0,    0,    0]], device='cuda:0'), 'start_positions': tensor([ 20,  90,  89,  41,  25,  96, 106, 132], device='cuda:0'), 'token_type_ids': tensor([[0, 0, 0,  ..., 0, 0, 0],
        [0, 0, 0,  ..., 0, 0, 0],
        [0, 0, 0,  ..., 0, 0, 0],
        ...,
        [0, 0, 0,  ..., 0, 0, 0],
        [0, 0, 0,  ..., 0, 0, 0],
        [0, 0, 0,  ..., 0, 0, 0]], device='cuda:0')}
```

```
# This function preprocesses a question answering dataset, tokenizing the question and context text
# and finding the right offsets for the answer spans in the tokenized context (to use as labels).
# Adapted from https://github.com/huggingface/transformers/blob/master/examples/pytorch/question-answering/run_qa.py
def prepare_train_dataset_qa(examples, tokenizer, max_seq_length=None):
    questions = [q.lstrip() for q in examples["question"]]
    max_seq_length = tokenizer.model_max_length
    # tokenize both questions and the corresponding context
    # if the context length is longer than max_length, we split it to several
    # chunks of max_length
    tokenized_examples = tokenizer(
        questions,
        examples["context"],
        truncation="only_second",
        max_length=max_seq_length,
        stride=min(max_seq_length // 2, 128),
        return_overflowing_tokens=True,
        return_offsets_mapping=True,
        padding="max_length"
    )

    # Since one example might give us several features if it has a long context,
    # we need a map from a feature to its corresponding example.
    sample_mapping = tokenized_examples.pop("overflow_to_sample_mapping")
    # The offset mappings will give us a map from token to character position
    # in the original context. This will help us compute the start_positions
    # and end_positions to get the final answer string.
    offset_mapping = tokenized_examples.pop("offset_mapping")

    tokenized_examples["start_positions"] = []
    tokenized_examples["end_positions"] = []

    tokenized_examples["example_id"] = []

    for i, offsets in enumerate(offset_mapping):
        input_ids = tokenized_examples["input_ids"][i]
        # We will label features not containing the answer the index of the CLS token.
        cls_index = input_ids.index(tokenizer.cls_token_id)
        sequence_ids = tokenized_examples.sequence_ids(i)
        # from the feature idx to sample idx
        sample_index = sample_mapping[i]
        # get the answer for a feature
        answers = examples["answers"][sample_index]

        tokenized_examples["example_id"].append(examples["id"][sample_index])

        if len(answers["answer_start"]) == 0:
            tokenized_examples["start_positions"].append(cls_index)
            tokenized_examples["end_positions"].append(cls_index)
        else:
            # Start/end character index of the answer in the text.
            start_char = answers["answer_start"][0]
            end_char = start_char + len(answers["text"][0])

            # Start token index of the current span in the text.
            token_start_index = 0
            while sequence_ids[token_start_index] != 1:
                token_start_index += 1

            # End token index of the current span in the text.
            token_end_index = len(input_ids) - 1
            while sequence_ids[token_end_index] != 1:
                token_end_index -= 1

            # Detect if the answer is out of the span (in which case this feature is labeled with the CLS index).
            if not (offsets[token_start_index][0] <= start_char and
                    offsets[token_end_index][1] >= end_char):
                tokenized_examples["start_positions"].append(cls_index)
                tokenized_examples["end_positions"].append(cls_index)
            else:
                # Otherwise move the token_start_index and token_end_index to the two ends of the answer.
                # Note: we could go after the last offset if the answer is the last word (edge case).
                while token_start_index < len(offsets) and \
                        offsets[token_start_index][0] <= start_char:
                    token_start_index += 1
                tokenized_examples["start_positions"].append(
                    token_start_index - 1)
                while offsets[token_end_index][1] >= end_char:
                    token_end_index -= 1
                tokenized_examples["end_positions"].append(token_end_index + 1)

    return tokenized_examples
```

_Originally posted by @yanllearnn in https://github.com/huggingface/datasets/issues/3333#issuecomment-983457161_


Make LABR dataset streamable

Fix LABR dataset to make it streamable.

Related to: #3350.


Add VCTK dataset

Hello @patrickvonplaten, I hope it's okay to ping you with a (dumb) question!

I've been trying to get `dl_manager.download_and_extract(_DL_URL)` to work with no avail. I verified that this is a problem on two different machines (lab server, GCP), so I doubt it's an issue with network connectivity. Here is the full trace.

```
(venv) (base) jaketae@jake-gpu1:~/documents/datasets$ datasets-cli test datasets/vctk --save_infos --all_configs
Testing builder 'main' (1/1)
Downloading and preparing dataset vctk/main to /home/jaketae/.cache/huggingface/datasets/vctk/main/0.9.2/2bfa52a93469fa9d6d4b1831c6511db5442b9f4e48620aef2bc3890d7a5268a8...
Traceback (most recent call last):
  File "/home/jaketae/documents/datasets/venv/bin/datasets-cli", line 33, in <module>
    sys.exit(load_entry_point('datasets', 'console_scripts', 'datasets-cli')())
  File "/home/jaketae/documents/datasets/src/datasets/commands/datasets_cli.py", line 33, in main
    service.run()
  File "/home/jaketae/documents/datasets/src/datasets/commands/test.py", line 146, in run
    builder.download_and_prepare(
  File "/home/jaketae/documents/datasets/src/datasets/builder.py", line 593, in download_and_prepare
    self._download_and_prepare(
  File "/home/jaketae/documents/datasets/src/datasets/builder.py", line 659, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/jaketae/.cache/huggingface/modules/datasets_modules/datasets/vctk/2bfa52a93469fa9d6d4b1831c6511db5442b9f4e48620aef2bc3890d7a5268a8/vctk.py", line 76, in _split_generators
    root_path = dl_manager.download_and_extract(_DL_URL)
  File "/home/jaketae/documents/datasets/src/datasets/utils/download_manager.py", line 283, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/home/jaketae/documents/datasets/src/datasets/utils/download_manager.py", line 195, in download
    downloaded_path_or_paths = map_nested(
  File "/home/jaketae/documents/datasets/src/datasets/utils/py_utils.py", line 234, in map_nested
    return function(data_struct)
  File "/home/jaketae/documents/datasets/src/datasets/utils/download_manager.py", line 216, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/home/jaketae/documents/datasets/src/datasets/utils/file_utils.py", line 298, in cached_path
    output_path = get_from_cache(
  File "/home/jaketae/documents/datasets/src/datasets/utils/file_utils.py", line 608, in get_from_cache
    raise ConnectionError(f"Couldn't reach {url}")
ConnectionError: Couldn't reach https://datashare.is.ed.ac.uk/bitstream/handle/10283/3443/VCTK-Corpus-0.92.zip
```

On my local, however, the URL correctly points to the download zip file. My admittedly naive guess is that the website is web-crawler or scraper proof (requiring specific headers, etc.), but I also think I might have just missed a very basic step in the process.

Apologies for the delayed PR, and TIA for the help!

Hey @jaketae,

Hmm, yeah I don't know really either - the link also works correctly for me when doing:

```
wget https://datashare.is.ed.ac.uk/bitstream/handle/10283/3443/VCTK-Corpus-0.92.zip
```

I think however that I had a similar problem previously with Edinburgh's (`.ed.ac.uk`) websites which I solved with the following hack. Not sure if this could be the same problem here...
https://github.com/huggingface/datasets/blob/e1104ad5d3e83f8b1571e0d6fef4fdabf0a1fde5/datasets/ami/ami.py#L364

The AMI dataset is stored under a different website though it seems: `"https://groups.inf.ed.ac.uk/ami/AMICorpusMirror//amicorpus/{}/audio/{}"`

so not 100p sure if this solves the problem

Hi @patrickvonplaten,

Thanks for the feedback! Sadly, disabling multi-processing didn't cut it for me.

I've been looking at VCTK code in [`torchaudio`](https://pytorch.org/audio/stable/_modules/torchaudio/datasets/vctk.html) and [`tfds`](https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/audio/vctk.py). I don't think they're using a hack to accomplish this, so I'll try to look into it to see if I can pinpoint the cause. I'll keep you in the loop here. Thank you!

Hi @patrickvonplaten,

After more investigation, I found that simply increasing `etag_timeout` in `get_from_cache` from 10 to 100 solved it. However, unless I'm missing something, an issue is that `etag_timeout` is basically hard-coded as a default parameter because `cached_path`, which calls `get_from_cache` has no way of modifying the default.

https://github.com/huggingface/datasets/blob/b25ac1d62670e7b339ed552ecc37846d2abd30c7/src/datasets/utils/file_utils.py#L298-L310

https://github.com/huggingface/datasets/blob/b25ac1d62670e7b339ed552ecc37846d2abd30c7/src/datasets/utils/file_utils.py#L497-L510

I can think of two solutions.

* Simply increase the default to 100
* Allow `etag_timeout` to be modifiable on a per-dataset basis by integrating it to `download_config` (maybe this is already supported?)

Thank you!

I think in this case we can increase the `etag_timeout` - what do you think @lhoestq @albertvillanova ?

Yes let's increase it to 100 for the moment. Later we can see if it really needed to move it into `download_config` or not

Thanks for the feedback @patrickvonplaten @lhoestq, I'll continue working on this in that direction!

Fixes #1837.


Avoid content-encoding issue while streaming datasets

This PR will fix streaming of datasets served with gzip content-encoding:
```
ClientPayloadError: 400, message='Can not decode content-encoding: gzip'
```

Fix #2918.

CC: @severo


raise exception instead of using assertions.

@mariosasko - Thanks for the review & suggestions. Updated as per the suggestions.

@mariosasko - Hello, Are there any additional changes required from my end??. Wondering if this PR can be merged or still pending on additional steps.

fix for the remaining files https://github.com/huggingface/datasets/issues/3171


BLEURT: Match key names to correspond with filename

Thanks for the suggestion! I think the current checked-in `CHECKPOINT_URLS` is already not working. I believe anyone who tried using the new ckpts (`BLEURT-20-X`) can't unless this fix is in. The zip file from bleurt side unzips to directory name matching the filename (capitalized for new ones). For example without current changes I'd get the following error

```
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-5-f6832fe20f84> in <module>()
      1 predictions = ["hello there", "general kenobi"]
      2 references = ["hello there", "general kenobi"]
----> 3 bleurt = datasets.load_metric("bleurt", "bleurt-20")
      4 results = bleurt.compute(predictions=predictions, references=references)

4 frames
/usr/local/lib/python3.7/dist-packages/bleurt/checkpoint.py in read_bleurt_config(path)
     84   """Reads and checks config file from a BLEURT checkpoint."""
     85   assert tf.io.gfile.exists(path), \
---> 86       "Could not find BLEURT checkpoint {}".format(path)
     87   config_path = os.path.join(path, CONFIG_FILE)
     88   assert tf.io.gfile.exists(config_path), \

AssertionError: Could not find BLEURT checkpoint /root/.cache/huggingface/metrics/bleurt/bleurt-20/downloads/extracted/e34c60f1a05394ecda54e253a10413ca7b5d59f9a23f3cc73258c6b78ffa2f50/bleurt-20
```
inspecting specified path I see that directory name is `BLEURT-20` instead of `bleurt-20`.
Other solution similar to your suggestion is meddle with `dl_manager.download_and_extract` to unzip to paths with lowering all the paths but I imagine this will affect other parts of the library.

Indeed, good catch ! Your solution that fixes `CHECKPOINT_URLS ` is simple and works well, thanks :)

Furthermore to avoid breaking changes though we could also keep the support for the lowercase one:
```python
        if self.config_name.lower() in CHECKPOINT_URLS:
            checkpoint_name = self.config_name.lower()
        elif self.config_name.upper() in CHECKPOINT_URLS:
            checkpoint_name = self.config_name.upper()
        else:
            raise KeyError(
                f"{self.config_name} model not found. You should supply the name of a model checkpoint for bleurt in {CHECKPOINT_URLS.keys()}"
            )
```
and then we can use `checkpoint_name` instead of `self.config_name` to download and instantiate the model:
```python
        model_path = dl_manager.download_and_extract(CHECKPOINT_URLS[checkpoint_name])
        self.scorer = score.BleurtScorer(os.path.join(model_path, checkpoint_name))
```

Please let me know if that sounds reasonable to you !

Thanks for the suggestion! I believe your suggestion should work to make keys case insensitive. Changes are committed to the PR now.

In order to properly locate downloaded ckpt files key name needs to match filename. Correcting change introduced in #3235


iter_archive for zip files

And also don't always try streaming with Google Drive - it can have issues because of how Google Drive works (with quotas, restrictions, etc.) and it can indeed cause `BlockSizeError`.

Feel free to host your test data elsewhere, such as in a dataset repository on https://huggingface.co (see [here](https://huggingface.co/docs/datasets/upload_dataset.html#upload-your-files) for a tutorial on how to upload files)

* In this PR, I added the option to iterate through zipfiles for `download_manager.py` only.
* Next PR will be the same applied to `streaming_download_manager.py`.
* Related issue #3272.
## Comments :
* There is no `.isreg()` equivalent in zipfile library to check if file is Regular so I used `.is_dir()` instead to skip directories.
* For now I got `streaming_download_manager.py`  working for local zip files, but not for urls. I get the following error when I test it on an archive in google drive, so still working on it.  `BlockSizeError: Got more bytes so far (>2112) than requested (22)`

## Tasks :
- [x] download_manager.py
- [ ] streaming_download_manager.py


Failed to convert `string` with pyarrow for QED since 1.15.0

Something isn't working

Scratch that, probably the old and incompatible usage of dataset builder from promptsource.

## Describe the bug
Loading QED was fine until 1.15.0.
related: bigscience-workshop/promptsource#659, bigscience-workshop/promptsource#670

Not sure where the root cause is, but here are some candidates:
- #3158
- #3120
- #3196
- #2891

## Steps to reproduce the bug
```python
load_dataset("qed")
```

## Expected results
Loading completed.

## Actual results
```shell
ArrowInvalid: Could not convert in with type str: tried to convert to boolean
Traceback:
File "/Users/s0s0cr3/Library/Python/3.9/lib/python/site-packages/streamlit/script_runner.py", line 354, in _run_script
    exec(code, module.__dict__)
File "/Users/s0s0cr3/Documents/GitHub/promptsource/promptsource/app.py", line 260, in <module>
    dataset = get_dataset(dataset_key, str(conf_option.name) if conf_option else None)
File "/Users/s0s0cr3/Library/Python/3.9/lib/python/site-packages/streamlit/caching.py", line 543, in wrapped_func
    return get_or_create_cached_value()
File "/Users/s0s0cr3/Library/Python/3.9/lib/python/site-packages/streamlit/caching.py", line 527, in get_or_create_cached_value
    return_value = func(*args, **kwargs)
File "/Users/s0s0cr3/Documents/GitHub/promptsource/promptsource/utils.py", line 49, in get_dataset
    builder_instance.download_and_prepare()
File "/Users/s0s0cr3/Library/Python/3.9/lib/python/site-packages/datasets/builder.py", line 607, in download_and_prepare
    self._download_and_prepare(
File "/Users/s0s0cr3/Library/Python/3.9/lib/python/site-packages/datasets/builder.py", line 697, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
File "/Users/s0s0cr3/Library/Python/3.9/lib/python/site-packages/datasets/builder.py", line 1106, in _prepare_split
    num_examples, num_bytes = writer.finalize()
File "/Users/s0s0cr3/Library/Python/3.9/lib/python/site-packages/datasets/arrow_writer.py", line 456, in finalize
    self.write_examples_on_file()
File "/Users/s0s0cr3/Library/Python/3.9/lib/python/site-packages/datasets/arrow_writer.py", line 325, in write_examples_on_file
    pa_array = pa.array(typed_sequence)
File "pyarrow/array.pxi", line 222, in pyarrow.lib.array
File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
File "/Users/s0s0cr3/Library/Python/3.9/lib/python/site-packages/datasets/arrow_writer.py", line 121, in __arrow_array__
    out = pa.array(cast_to_python_objects(self.data, only_1d_for_numpy=True), type=type)
File "pyarrow/array.pxi", line 305, in pyarrow.lib.array
File "pyarrow/array.pxi", line 39, in pyarrow.lib._sequence_to_array
File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.15.0, 1.16.1
- Platform: macOS 1.15.7 or above
- Python version: 3.7.12 and 3.9
- PyArrow version: 3.0.0, 5.0.0, 6.0.1


Failed to download species_800 from Google Drive zip file

Something isn't working

Hi,

the dataset is downloaded normally on my machine. Maybe the URL was down at the time of your download. Could you try again?

> Hi,
>
> the dataset is downloaded normally on my machine. Maybe the URL was down at the time of your download. Could you try again?

I have tried that many times with both load_dataset() and a browser almost simultaneously. The browser always works for me while load_dataset() fails.

@mariosasko
> the dataset is downloaded normally on my machine. Maybe the URL was down at the time of your download. Could you try again?

I've tried yet again just a moment ago. This time I realize that, the step `(... post-processed: Unknown size, total: 20.89 MiB) to /Users/mike/.cache/huggingface/datasets/species800/species_800/1.0.0/532167f0bb8fbc0d77d6d03c4fd642c8c55527b9c5f2b1da77f3d00b0e559976...` and the one after seem unstable. If I want to retry, I will have to delete it (and probably other cache lock files). It **_sometimes_** works.

But I didn't try `download_mode="force_redownload"` yet.

Anyway, I suppose this isn't really a pressing issue for the time being, so I'm going to close this. Thank you.

## Describe the bug
One can manually download the zip file on Google Drive, but `load_dataset()` cannot.
related: #3248

## Steps to reproduce the bug
```shell
> python
Python 3.7.12 (default, Sep  5 2021, 08:34:29)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
```
```python
>>> from datasets import load_dataset
>>> s800 = load_dataset("species_800")
```

## Expected results
species_800 downloaded.

## Actual results
```shell
Downloading: 5.68kB [00:00, 1.22MB/s]
Downloading: 2.70kB [00:00, 691kB/s]
Downloading and preparing dataset species800/species_800 (download: 17.36 MiB, generated: 3.53 MiB, post-processed: Unknown size, total: 20.89 MiB) to /Users/mike/.cache/huggingface/datasets/species800/species_800/1.0.0/532167f0bb8fbc0d77d6d03c4fd642c8c55527b9c5f2b1da77f3d00b0e559976...
  0%|                                                                                | 0/1 [00:00<?, ?it/s]Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/load.py", line 1632, in load_dataset
    use_auth_token=use_auth_token,
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/builder.py", line 608, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/builder.py", line 675, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/Users/mike/.cache/huggingface/modules/datasets_modules/datasets/species_800/532167f0bb8fbc0d77d6d03c4fd642c8c55527b9c5f2b1da77f3d00b0e559976/species_800.py", line 104, in _split_generators
    downloaded_files = dl_manager.download_and_extract(urls_to_download)
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 284, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 197, in download
    download_func, url_or_urls, map_tuple=True, num_proc=download_config.num_proc, disable_tqdm=False
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/utils/py_utils.py", line 209, in map_nested
    for obj in utils.tqdm(iterable, disable=disable_tqdm)
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/utils/py_utils.py", line 209, in <listcomp>
    for obj in utils.tqdm(iterable, disable=disable_tqdm)
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/utils/py_utils.py", line 143, in _single_map_nested
    return function(data_struct)
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 217, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 305, in cached_path
    use_auth_token=download_config.use_auth_token,
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 594, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://drive.google.com/u/0/uc?id=1OletxmPYNkz2ltOr9pyT0b0iBtUWxslh&export=download/
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.14,0 1.15.0, 1.16.1
- Platform: macOS Catalina 10.15.7
- Python version: 3.7.12
- PyArrow version: 6.0.1


Add ArrayXD docs

Documents support for dynamic first dimension in `ArrayXD` from #2891, and explain the `ArrayXD` feature in general.

Let me know if I'm missing anything @lhoestq :)


Better error message when download fails

From our discussions in https://github.com/huggingface/datasets/issues/3269 and https://github.com/huggingface/datasets/issues/3282 it would be nice to have better messages if a download fails.

In particular the error now shows:
- the error from the HEAD request if there's one
- otherwise the response code of the HEAD request

I also added an error to tell users to pass `use_auth_token` when the Hugging Face Hub returns 401 (Unauthorized).

While paying around with this I also fixed a minor issue with the `force_download` parameter that was not always taken into account


Fix ASSET dataset data URLs

> Hi @tianjianjiang, thanks for the fix.
> The links should also be updated in the `dataset_infos.json` file.
> The failing tests are due to the missing tag in the header of the `README.md` file:

Hi @albertvillanova, thank you for the info! My apologies for the messy PR.

Change the branch name "master" to "main" in the data URLs, since facebookresearch has changed that.


Mirror the canonical datasets to the Hugging Face Hub

enhancement

New feature or request

I created a GitHub project to keep track of what needs to be done:
https://github.com/huggingface/datasets/projects/3

I also store my code in a (private for now) repository at https://github.com/huggingface/mirror_canonical_datasets_on_hub

- [ ] create a repo on https://hf.co/datasets for every canonical dataset
- [ ] on every commit related to a dataset, update the hf.co repo

See https://github.com/huggingface/moon-landing/pull/1562

@SBrandeis: I let you edit this description if needed to precise the intent.


Fix JSON ClassLabel casting for integers

Loading a JSON dataset with ClassLabel feature types currently fails if the JSON data already has integers. Indeed currently it tries to convert the strings to integers without even checking if the data are not integers already.

For example this currently fails:
```python
from datasets import load_dataset, Features, ClassLabel

path = "data.json"
f = Features({"a": ClassLabel(names=["neg", "pos"])})
d = load_dataset("json", data_files=path, features=f)
```
data.json
```json
{"a": 0}
{"a": 1}
```

I fixed that by adding a line that checks the type of the JSON data before trying to convert them

cc @albertvillanova let me know if it sounds good to you


to_tf_dataset fails on TPU

bug

Something isn't working

This might be related to https://github.com/tensorflow/tensorflow/issues/38762 , what do you think @Rocketknight1 ?
> Dataset.from_generator is expected to not work with TPUs as it uses py_function underneath which is incompatible with Cloud TPU 2VM setup. If you would like to read from large datasets, maybe try to materialize it on disk and use TFRecordDataest instead.

Hi @lhoestq @nbroad1881, I think it's very similar, yes. Unfortunately `to_tf_dataset` uses `tf.numpy_function` which can't be compiled - this is a necessary evil to load from the underlying Arrow dataset. We need to update the notebooks/examples to clarify that this won't work, or to identify a workaround. You may be able to get it to work on an actual cloud TPU VM, but those are quite new and we haven't tested it yet.

Thank you for the explanation. I didn't realize the nuances of `tf.numpy_function`. In this scenario, would it be better to use `export(format='tfrecord')` ?  It's not quite the same, but for very large datasets that don't fit in memory it looks like it is the only option. I haven't used `export` before, but I do recall reading that there are suggestions for how big and how many tfrecords there should be to not bottleneck the TPU. It might be nice if there were a way for the `export` method to split the files up into appropriate chunk sizes depending on the size of the dataset and the number of devices. And if that is too much, it would be nice to be able to specify the number of files that would be created when using `export`. Well... maybe the user should just do the chunking themselves and call `export` a bunch of times.  Whatever the case, you have been helpful. Thanks Tensorflow boy ;-)

Yeah, this is something we really should have a proper guide on. I'll make a note to test some things and make a 'TF TPU best practices' notebook at some point, but in the meantime I think your solution of exporting TFRecords will probably work.

Using `to_tf_dataset` to create a dataset and then putting it in `model.fit` results in an internal error on TPUs. I've only tried on Colab and Kaggle TPUs, not GCP TPUs.

## Steps to reproduce the bug
I made a colab to show the error. https://colab.research.google.com/drive/12x_PFKzGouFxqD4OuWfnycW_1TaT276z?usp=sharing

## Expected results
dataset from `to_tf_dataset` works in `model.fit`
Right below the first error in the colab I use `tf.data.Dataset.from_tensor_slices` and `model.fit` works just fine. This is the desired outcome.

## Actual results
```
InternalError: 5 root error(s) found.
  (0) INTERNAL: {{function_node __inference_train_function_30558}} failed to connect to all addresses
Additional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:
:{"created":"@1638231897.932218653","description":"Failed to pick subchannel","file":"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3151,"referenced_errors":[{"created":"@1638231897.932216754","description":"failed to connect to all addresses","file":"third_party/grpc/src/core/lib/transport/error_utils.cc","file_line":161,"grpc_status":14}]}
	 [[{{node StatefulPartitionedCall}}]]
	 [[MultiDeviceIteratorGetNextFromShard]]
Executing non-communication op <MultiDeviceIteratorGetNextFromShard> originally returned UnavailableError, and was replaced by InternalError to avoid invoking TF network error handling logic.
	 [[RemoteCall]]
	 [[IteratorGetNextAsOptional]]
	 [[tpu_compile_succeeded_assert/_14023832043698465348/_7/_439]]
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.16.1
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 3.0.0
- Tensorflow 2.7.0
- `transformers` 4.12.5


[WIP] Add doctests for tutorials

I manage to remove the mentions of ellipsis in the code by launching the command as follows:

```
python -m doctest -v docs/source/load_hub.rst -o=ELLIPSIS
```

The way you put your ellipsis will only work on mac, I've adapted it for linux as well with the following:

```diff
    >>> from datasets import load_dataset_builder
    >>> dataset_builder = load_dataset_builder('imdb')
-   >>> print(dataset_builder.cache_dir) #doctest: +ELLIPSIS
-   /Users/.../.cache/huggingface/datasets/imdb/plain_text/1.0.0/...
+   >>> print(dataset_builder.cache_dir)
+   /.../.cache/huggingface/datasets/imdb/plain_text/1.0.0/...
```

This passes on my machine:

```
Trying:
    print(dataset_builder.cache_dir)
Expecting:
    /.../.cache/huggingface/datasets/imdb/plain_text/1.0.0/...
ok
```

I'm getting a last error:

```py
Expected:
    DatasetDict({
        train: Dataset({
            features: ['sentence1', 'sentence2', 'label', 'idx'],
            num_rows: 3668
        })
        validation: Dataset({
            features: ['sentence1', 'sentence2', 'label', 'idx'],
            num_rows: 408
        })
        test: Dataset({
            features: ['sentence1', 'sentence2', 'label', 'idx'],
            num_rows: 1725
        })
    })
Got:
    DatasetDict({
        train: Dataset({
            features: ['idx', 'label', 'sentence1', 'sentence2'],
            num_rows: 3668
        })
        validation: Dataset({
            features: ['idx', 'label', 'sentence1', 'sentence2'],
            num_rows: 408
        })
        test: Dataset({
            features: ['idx', 'label', 'sentence1', 'sentence2'],
            num_rows: 1725
        })
    })
```

But this is due to `doctest` looking for an exact match and the list having an unordered print order. I wish `doctest` would be a bit more flexible with that.

Opening a PR as discussed with @LysandreJik for some help with doctest issues. The goal is to add doctests for each of the tutorials in the documentation to make sure the code samples work as shown.

### Issues

A doctest has been added in the docstring of the `load_dataset_builder` function in `load.py` to handle variable outputs with the `ELLIPSIS` directive. When I run doctest on the `load_hub.rst` file, doctest should recognize the expected output from the docstring, and the corresponding code sample in `load_hub.rst` should pass. I am having the same issue with handling tracebacks in the `load_dataset` function.

From the docstring:
```
>>> dataset_builder.cache_dir #doctest: +ELLIPSIS
/Users/.../.cache/huggingface/datasets/imdb/plain_text/1.0.0/...
```
Test result:
```
Failed example:
    dataset_builder.cache_dir
Expected:
    /Users/.../.cache/huggingface/datasets/imdb/plain_text/1.0.0/...
Got:
    /Users/steven/.cache/huggingface/datasets/imdb/plain_text/1.0.0/2fdd8b9bcadd6e7055e742a706876ba43f19faee861df134affd7a3f60fc38a1
```

I am able to get the doctest to pass by adding the doctest directives (`ELLIPSIS` and `NORMALIZE_WHITESPACE`) to the code samples in the `rst` file directly. But my understanding is that these directives should also work in the docstrings of the functions. I am running the test from the root of the directory:

```
python -m doctest -v docs/source/load_hub.rst
```


Typing of Dataset.__getitem__ could be improved.

Something isn't working

Hi ! Thanks for the suggestion, I didn't know about this decorator.

If you are interesting in contributing, feel free to open a pull request to add the overload methods for each typing combination :) To assign you to this issue, you can comment `#self-assign` in this thread.

`Dataset.__getitem__` is defined right here: https://github.com/huggingface/datasets/blob/e6f1352fe19679de897f3d962e616936a17094f5/src/datasets/arrow_dataset.py#L1840

## Describe the bug

The newly added typing for Dataset.__getitem__ is Union[Dict, List]. This makes tools like mypy a bit awkward to use as we need to check the type manually. We could use type overloading to make this easier. [Documentation](https://docs.python.org/3/library/typing.html#typing.overload)

## Steps to reproduce the bug
Let's have a file `test.py`

```python
from typing import List, Dict, Any

from datasets import Dataset

ds = Dataset.from_dict({
    'a': [1,2,3],
    'b': ["1", "2", "3"]
})

one_colum: List[str] = ds['a']
some_index: Dict[Any, Any] = ds[1]
```

## Expected results

Running `mypy test.py` should not give any error.

## Actual results

```
test.py:10: error: Incompatible types in assignment (expression has type "Union[Dict[Any, Any], List[Any]]", variable has type "List[str]")
test.py:11: error: Incompatible types in assignment (expression has type "Union[Dict[Any, Any], List[Any]]", variable has type "Dict[Any, Any]")
Found 2 errors in 1 file (checked 1 source file)
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.13.3
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.8
- PyArrow version: 6.0.1


Add support for multiple dynamic dimensions and to_pandas conversion for dynamic arrays

Add support for multiple dynamic dimensions (e.g. `(None, None, 3)` for arbitrary sized images) and `to_pandas()` conversion for dynamic arrays.

TODOs:
* [ ] Cleaner code
* [ ] Formatting issues (if NumPy doesn't allow broadcasting even though dtype is np.object)
* [ ] Fix some issues with zero-dim tensors
* [ ] Tests


add Speech commands dataset

@lhoestq
Hi Quentin! Thank you for your feedback and suggestions! 🤗

Yes, that was actually what I wanted to do next - I mean the steaming stuff :)
Also, I need to make some changes to the readme (to account for the updated features set).

Hopefully, I will be done by tomorrow afternoon if that's ok.

@lhoestq  Hi Quentin!

I've implemented (hopefully, correctly) the streaming compatibility but the problem with the current approach is that we first need to iterate over the full archive anyway to get the list of filenames for train and validation sets (see [this](https://github.com/huggingface/datasets/pull/3335/files#diff-aeea540d136025e30a842856779e9c6485a5dc6fc9eb7fd6d3be2acd2f49b8e3R186), the same approach is implemented in TFDS version). Only after that, we can generate examples, so we cannot stream the dataset before the first iteration ends and it takes some time. It's probably not the most effective way.

If the streaming mode is turned off, this approach (with two iterations) is actually slower than the previous implementation (with archive extraction).

My suggestion is to host separate archives for each split prepared in advance. That way there would be no need for iterating over the common archive to collect train and validation filenames.  @anton-l  suggested to make AWS mirrors for them. I've prepared these archives, for now you can take a look at them [here](https://drive.google.com/drive/folders/1oMrZHzPgHAKprKJuvih91CM8KMSzh_pL?usp=sharing). I simplified their structure a bit so if we switch to using them, the code then should be changed (and simplified) a bit too.

Hi ! Thanks for the changes :)

> My suggestion is to host separate archives for each split prepared in advance. That way there would be no need for iterating over the common archive to collect train and validation filenames. @anton-l suggested to make AWS mirrors for them. I've prepared these archives, for now you can take a look at them here. I simplified their structure a bit so if we switch to using them, the code then should be changed (and simplified) a bit too.

I agree, I just uploaded them on AWS

https://s3.amazonaws.com/datasets.huggingface.co/SpeechCommands/v0.01/v0.01_test.tar.gz
https://s3.amazonaws.com/datasets.huggingface.co/SpeechCommands/v0.01/v0.01_train.tar.gz
https://s3.amazonaws.com/datasets.huggingface.co/SpeechCommands/v0.01/v0.01_validation.tar.gz
https://s3.amazonaws.com/datasets.huggingface.co/SpeechCommands/v0.02/v0.02_test.tar.gz
https://s3.amazonaws.com/datasets.huggingface.co/SpeechCommands/v0.02/v0.02_validation.tar.gz

Note that in the future we can move those files to actual repositories on the Hugging Face Hub, since we are migrating the datasets from this repository to the Hugging Face Hub (as mirrors), to make them more accessible to the community.

@lhoestq Thank you! Gonna look at this tomorrow :)

@lhoestq I've modified the code to fit new data format, now it works for v0.01 but doesn't work for v0.02 as the training archive is missing. Could you please create a mirror for that one too? You can find it [here](https://drive.google.com/file/d/1mPjnVMYb-VhPprGlOX8v9TBT1GT-rtcp/view?usp=sharing)

And when it's done I'll need to regenerate all the meta / dummy stuff, and this version will be ready for a review :)

closes  #3283


Integrate Polars library

enhancement

New feature or request

If possible, a neat API could be something like `Dataset.to_polars()`, as well as `Dataset.set_format("polars")`

Note they use a "custom" implementation of Arrow: [Arrow2](https://github.com/jorgecarleitao/arrow2).

Check potential integration of the Polars library: https://github.com/pola-rs/polars
- Benchmark: https://h2oai.github.io/db-benchmark/

CC: @thomwolf @lewtun