`SystemError 15` thrown in `Dataset.__del__` when using `Dataset.map()` with `num_proc>1`
NB: even if the error is raised, the dataset is successfully cached. So restarting the script after every `map()` allows to ultimately run the whole preprocessing. But this prevents to realistically run the code over multiple nodes. Hi,

It's not easy to debug the problem without the script. I may be wrong since I'm not very familiar with PyTorch Lightning, but shouldn't you preprocess the data in the `prepare_data` function of `LightningDataModule` and not in the `setup` function.
As you can't modify the module state in `prepare_data` (according to the docs), use the `cache_file_name` argument in `Dataset.map` there, and reload the processed data in `setup` with `Dataset.from_file(cache_file_name)`. If `num_proc>1`, check the docs on the `suffix_template` argument of `Dataset.map` to get an idea what the final `cache_file_names` are going to be.

Let me know if this helps. Hi @mariosasko, thank you for the hint, that helped me to move forward with that issue. 

I did a major refactoring of my project to disentangle my `LightningDataModule` and `Dataset`.  Just FYI, it looks like:

```python
class Builder():
   def __call__() -> DatasetDict:
      # load and preprocess the data
      return dataset

class DataModule(LightningDataModule):
    def prepare_data():
       self.builder()
    def setup():
       self.dataset = self.builder()
```

Unfortunately, the entanglement between `LightningDataModule` and `Dataset` was not the issue.

The culprit was `hydra` and a slight adjustment of the structure of my project solved this issue. The problematic project  structure was:

```
src/
 | - cli.py
 | - training/
      | -experiment.py

# code in experiment.py
def run_experiment(config):
     # preprocess data and run
     
# code in cli.py
@hydra.main(...)
def run(config):
    return run_experiment(config)
```

Moving `run()` from `clip.py` to `training.experiment.py` solved the issue with `SystemError 15`. No idea why. 

Even if the traceback was referring to `Dataset.__del__`, the problem does not seem to be primarily related to `datasets`, so I will close this issue. Thank you for your help!
## Describe the bug
I use `datasets.map` to preprocess some data in my application. The error `SystemError 15` is thrown at the end of the execution of `Dataset.map()` (only with `num_proc>1`. Traceback included bellow. 
 
The exception is raised only when the code runs within a specific context. Despite ~10h spent investigating this issue, I have failed to isolate the bug, so let me describe my setup. 

In my project, `Dataset` is wrapped into a `LightningDataModule` and the data is preprocessed when calling `LightningDataModule.setup()`. Calling `.setup()` in an isolated script works fine (even when wrapped with `hydra.main()`). However, when calling `.setup()` within the experiment script (depends on `pytorch_lightning`), the script crashes and `SystemError 15`.

I could avoid throwing this error by modifying ` Dataset.__del__()` (see bellow), but I believe this only moves the problem somewhere else. I am completely stuck with this issue, any hint would be welcome. 

```python
class Dataset()
    ...
    def __del__(self):
        if hasattr(self, "_data"):
            _ = self._data # <- ugly trick that allows avoiding the issue.
            del self._data
        if hasattr(self, "_indices"):
            del self._indices
```

## Steps to reproduce the bug
```python
# Unfortunately I couldn't isolate the bug.
```

## Expected results
Calling `Dataset.map()` without throwing an exception. Or at least raising a more detailed exception/traceback.

## Actual results
```
Exception ignored in: <function Dataset.__del__ at 0x7f7cec179160>███████████████████████████████████████████████████| 5/5 [00:05<00:00,  1.17ba/s]
Traceback (most recent call last):
  File ".../python3.8/site-packages/datasets/arrow_dataset.py", line 906, in __del__
    del self._data
  File ".../python3.8/site-packages/ray/worker.py", line 1033, in sigterm_handler
    sys.exit(signum)
SystemExit: 15

```

## Environment info

Tested on 2 environments:

**Environment 1.**
- `datasets` version: 1.14.0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.8
- PyArrow version: 6.0.0

**Environment 2.**
- `datasets` version: 1.14.0
- Platform: Linux-4.18.0-305.19.1.el8_4.x86_64-x86_64-with-glibc2.28
- Python version: 3.9.7
- PyArrow version: 6.0.0
