A description of this dataset can be found at https://arxiv.org/abs/2305.07759

Copied from roneneldan/TinyStories

Modified with:

```
import ftfy.bad_codecs
from datasets import Dataset, DatasetDict

train = open('./TinyStories-train.txt', 'r', encoding='sloppy-windows-1252').read()
train = train.split('<|endoftext|>')
train = [l.strip() for l in train]

valid = open('./TinyStories-valid.txt', 'r', encoding='sloppy-windows-1252').read()
valid = valid.split('<|endoftext|>')
valid = [l.strip() for l in valid]

dataset = DatasetDict({
    'train': Dataset.from_dict({'text': train }),
    'validation': Dataset.from_dict({'text': valid}),
})
dataset.save_to_disk('./TinyStories')
```
arxiv:2305.07759 region:us
{"dataset_info": {"features": [{"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1911420483, "num_examples": 2119719}, {"name": "validation", "num_bytes": 19306310, "num_examples": 21990}], "download_size": 1000775442, "dataset_size": 1930726793}}
TAGS
#arxiv-2305.07759 #region-us 

A description of this dataset can be found at URL

Copied from roneneldan/TinyStories

Modified with:


TAGS
#arxiv-2305.07759 #region-us 
