region:us
---
dataset_info:
  features:
  - name: id
    dtype: string
  - name: text
    dtype: string
  - name: dataset_id
    dtype: string
  splits:
  - name: train
    num_bytes: 124374684398.0
    num_examples: 31661477
  download_size: 67451668212
  dataset_size: 124374684398.0
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
---
# Dataset Card for "tok-corpus-shuffled"

This is the dataset used to fit custom tokenizers. Goal is to have a ytokenizer that is good for French, Englsih and Code.
The dataset uploaded is shuffled to facilitate subsampling it for tokenizer training.

```
French
Dataset({
    features: ['id', 'text', 'dataset_id'],
    num_rows: 16881941
})
Code
Dataset({
    features: ['id', 'text', 'dataset_id'],
    num_rows: 6338566
})
English
Dataset({
    features: ['text', 'id', 'dataset_id'],
    num_rows: 8440970
})
Size of Concatenated: 124.0 GB
Size of French: 58.0 GB, ratio of 0.4705131689639972
Size of Code: 28.0 GB, ratio of 0.23046591420706297
Size of English: 37.0 GB, ratio of 0.29902091682893983
```