All of the data together is around 41GB. It's the last hidden states of 131,072 samples from refinedweb padded/truncated to 512 tokens on the left, fed through [google/flan-t5-small](https://hf.co/google/flan-t5-small).

Structure:

```
{
  "encoding": List, shaped (512, 512) aka (tokens, d_model),
  "text": String, the original text that was encoded,
  "attention_mask": List, binary mask to pass to your model with encoding to not attend to pad tokens
}
```

just a tip, you cannot load this with the RAM in the free ver of google colab, not even a single file, streaming won't work either. I have 80gb RAM and it was barely enough to work with streaming.
task_categories:feature-extraction size_categories:100K<n<1M language:en license:apache-2.0 t5 flan region:us
{"language": ["en"], "license": "apache-2.0", "size_categories": ["100K<n<1M"], "task_categories": ["feature-extraction"], "tags": ["t5", "flan"]}
TAGS
#task_categories-feature-extraction #size_categories-100K<n<1M #language-English #license-apache-2.0 #t5 #flan #region-us 

All of the data together is around 41GB. It's the last hidden states of 131,072 samples from refinedweb padded/truncated to 512 tokens on the left, fed through google/flan-t5-small.

Structure:



just a tip, you cannot load this with the RAM in the free ver of google colab, not even a single file, streaming won't work either. I have 80gb RAM and it was barely enough to work with streaming.
TAGS
#task_categories-feature-extraction #size_categories-100K<n<1M #language-English #license-apache-2.0 #t5 #flan #region-us 

passage: TAGS
#task_categories-feature-extraction #size_categories-100K<n<1M #language-English #license-apache-2.0 #t5 #flan #region-us 
