# Dataset Card for the human reference genome

## Dataset Description

- **Repository:** [Nucleotide Transformer](https://github.com/instadeepai/nucleotide-transformer)
- **Paper:** [The Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics](https://www.biorxiv.org/content/10.1101/2023.01.11.523679v1)

### Dataset Summary

The Human reference genome dataset was constructed by considering all autosomal and sex chromosomes sequences from reference assembly [GRCh38/hg38](https://www.ncbi.nlm.nih.gov/assembly/GCF_000001405.26) and reaches a total of 3.2 billion nucleotides.

### Supported Tasks and Leaderboards

This dataset has been used as a pre-training corpus for the Nucleotide Transformers models. Depending on the configuration used, each sequence is 6,200 or 12,200 base pase pairs long. If the dataset is iterated without being shuffled, the first 100 nucleotides of a sequence are the same as the last 100 base pairs of the previous sequence, and the last 100 nucleotides are the same as the first 100 base pairs of the next sequence. During training, this allows for randomly selecting a nucleotide between the first 200 nucleotides of the sequence and start the tokenization from this nucleotide. That way, all the chromosome is covered and the model sees different tokens for a given sequence at each epoch. 

### Languages

DNA

## Dataset Structure

[N/A]


### Data Instances

For each instance, there is a string representing the sequence, a string indicating the chromosome, and two integers representing the index of the first and last nucleotide respectively. An instance is shown below:
```python
{'sequence': 'CATCTGCAGGTGTCTGACTTCCAGCAACTGCTGGCCTGTGCCAGGGTGCAAGCTGAGCACTGGAGTGGAGTTTTCCTGTGGAGAGGAGCCATGCCTAGAGTGGGATGGGCCATTGTTCATCTTCTGGCCCCTGTTGTCTGCATGTAACTTAATACCACAACCAGGCATAGGGGAAAGATTGGAGGAAAGATGAGTGAGAGCATCAACTTCTCTCACAACCTAGGCCAGTAAGTAGTGCTTGTGCTCATCTCCTTGGCTGTGATACGTGGCCGGCCCTCGCTCCAGCAGCTGGACCCCTACCTGCCGTCTGCTGCCATCGGAGCCCAAAGCCGGGCTGTGACTGCTCAGACCAGCCGGCTGGAGGGAGGGGCTCAGCAGGTCTGGCTTTGGCCCTGGGAGAGCAGGTGGAAGATCAGGCAGGCCATCGCTGCCACAGAACCCAGTGGATTGGCCTAGGTGGGATCTCTGAGCTCAACAAGCCCTCTCTGGGTGGTAGGTGCAGAGACGGGAGGGGCAGAGCCGCAGGCACAGCCAAGAGGGCTGAAGAAATGGTAGAACGGAGCAGCTGGTGATGTGTGGGCCCACCGGCCCCAGGCTCCTGTCTCCCCCCAGGTGTGTGGTGATGCCAGGCATGCCCTTCCCCAGCATCAGGTCTCCAGAGCTGCAGAAGACGACGGCCGACTTGGATCACACTCTTGTGAGTGTCCCCAGTGTTGCAGAGGTGAGAGGAGAGTAGACAGTGAGTGGGAGTGGCGTCGCCCCTAGGGCTCTACGGGGCCGGCGTCTCCTGTCTCCTGGAGAGGCTTCGATGCCCCTCCACACCCTCTTGATCTTCCCTGTGATGTCATCTGGAGCCCTGCTGCTTGCGGTGGCCTATAAAGCCTCCTAGTCTGGCTCCAAGGCCTGGCAGAGTCTTTCCCAGGGAAAGCTACAAGCAGCAAACAGTCTGCATGGGTCATCCCCTTCACTCCCAGCTCAGAGCCCAGGCCAGGGGCCCCCAAGAAAGGCTCTGGTGGAGAACCTGTGCATGAAGGCTGTCAACCAGTCCATAGGCAAGCCTGGCTGCCTCCAGCTGGGTCGACAGACAGGGGCTGGAGAAGGGGAGAAGAGGAAAGTGAGGTTGCCTGCCCTGTCTCCTACCTGAGGCTGAGGAAGGAGAAGGGGATGCACTGTTGGGGAGGCAGCTGTAACTCAAAGCCTTAGCCTCTGTTCCCACGAAGGCAGGGCCATCAGGCACCAAAGGGATTCTGCCAGCATAGTGCTCCTGGACCAGTGATACACCCGGCACCCTGTCCTGGACACGCTGTTGGCCTGGATCTGAGCCCTGGTGGAGGTCAAAGCCACCTTTGGTTCTGCCATTGCTGCTGTGTGGAAGTTCACTCCTGCCTTTTCCTTTCCCTAGAGCCTCCACCACCCCGAGATCACATTTCTCACTGCCTTTTGTCTGCCCAGTTTCACCAGAAGTAGGCCTCTTCCTGACAGGCAGCTGCACCACTGCCTGGCGCTGTGCCCTTCCTTTGCTCTGCCCGCTGGAGACGGTGTTTGTCATGGGCCTGGTCTGCAGGGATCCTGCTACAAAGGTGAAACCCAGGAGAGTGTGGAGTCCAGAGTGTTGCCAGGACCCAGGCACAGGCATTAGTGCCCGTTGGAGAAAACAGGGGAATCCCGAAGAAATGGTGGGTCCTGGCCATCCGTGAGATCTTCCCAGGGCAGCTCCCCTCTGTGGAATCCAATCTGTCTTCCATCCTGCGTGGCCGAGGGCCAGGCTTCTCACTGGGCCTCTGCAGGAGGCTGCCATTTGTCCTGCCCACCTTCTTAGAAGCGAGACGGAGCAGACCCATCTGCTACTGCCCTTTCTATAATAACTAAAGTTAGCTGCCCTGGACTATTCACCCCCTAGTCTCAATTTAAGAAGATCCCCATGGCCACAGGGCCCCTGCCTGGGGGCTTGTCACCTCCCCCACCTTCTTCCTGAGTCATTCCTGCAGCCTTGCTCCCTAACCTGCCCCACAGCCTTGCCTGGATTTCTATCTCCCTGGCTTGGTGCCAGTTCCTCCAAGTCGATGGCACCTCCCTCCCTCTCAACCACTTGAGCAAACTCCAAGACATCTTCTACCCCAACACCAGCAATTGTGCCAAGGGCCATTAGGCTCTCAGCATGACTATTTTTAGAGACCCCGTGTCTGTCACTGAAACCTTTTTTGTGGGAGACTATTCCTCCCATCTGCAACAGCTGCCCCTGCTGACTGCCCTTCTCTCCTCCCTCTCATCCCAGAGAAACAGGTCAGCTGGGAGCTTCTGCCCCCACTGCCTAGGGACCAACAGGGGCAGGAGGCAGTCACTGACCCCGAGACGTTTGCATCCTGCACAGCTAGAGATCCTTTATTAAAAGCACACTGTTGGTTTCTGCTCAGTTCTTTATTGATTGGTGTGCCGTTTTCTCTGGAAGCCTCTTAAGAACACAGTGGCGCAGGCTGGGTGGAGCCGTCCCCCCATGGAGCACAGGCAGACAGAAGTCCCCGCCCCAGCTGTGTGGCCTCAAGCCAGCCTTCCGCTCCTTGAAGCTGGTCTCCACACAGTGCTGGTTCCGTCACCCCCTCCCAAGGAAGTAGGTCTGAGCAGCTTGTCCTGGCTGTGTCCATGTCAGAGCAACGGCCCAAGTCTGGGTCTGGGGGGGAAGGTGTCATGGAGCCCCCTACGATTCCCAGTCGTCCTCGTCCTCCTCTGCCTGTGGCTGCTGCGGTGGCGGCAGAGGAGGGATGGAGTCTGACACGCGGGCAAAGGCTCCTCCGGGCCCCTCACCAGCCCCAGGTCCTTTCCCAGAGATGCCTGGAGGGAAAAGGCTGAGTGAGGGTGGTTGGTGGGAAACCCTGGTTCCCCCAGCCCCCGGAGACTTAAATACAGGAAGAAAAAGGCAGGACAGAATTACAAGGTGCTGGCCCAGGGCGGGCAGCGGCCCTGCCTCCTACCCTTGCGCCTCATGACCAGCTTGTTGAAGAGATCCGACATCAAGTGCCCACCTTGGCTCGTGGCTCTCACTGCAACGGGAAAGCCACAGACTGGGGTGAAGAGTTCAGTCACATGCGACCGGTGACTCCCTGTCCCCACCCCCATGACACTCCCCAGCCCTCCAAGGCCACTGTGTTTCCCAGTTAGCTCAGAGCCTCAGTCGATCCCTGACCCAGCACCGGGCACTGATGAGACAGCGGCTGTTTGAGGAGCCACCTCCCAGCCACCTCGGGGCCAGGGCCAGGGTGTGCAGCACCACTGTACAATGGGGAAACTGGCCCAGAGAGGTGAGGCAGCTTGCCTGGGGTCACAGAGCAAGGCAAAAGCAGCGCTGGGTACAAGCTCAAAACCATAGTGCCCAGGGCACTGCCGCTGCAGGCGCAGGCATCGCATCACACCAGTGTCTGCGTTCACAGCAGGCATCATCAGTAGCCTCCAGAGGCCTCAGGTCCAGTCTCTAAAAATATCTCAGGAGGCTGCAGTGGCTGACCATTGCCTTGGACCGCTCTTGGCAGTCGAAGAAGATTCTCCTGTCAGTTTGAGCTGGGTGAGCTTAGAGAGGAAAGCTCCACTATGGCTCCCAAACCAGGAAGGAGCCATAGCCCAGGCAGGAGGGCTGAGGACCTCTGGTGGCGGCCCAGGGCTTCCAGCATGTGCCCTAGGGGAAGCAGGGGCCAGCTGGCAAGAGCAGGGGGTGGGCAGAAAGCACCCGGTGGACTCAGGGCTGGAGGGGAGGAGGCGATCTTGCCCAAGGCCCTCCGACTGCAAGCTCCAGGGCCCGCTCACCTTGCTCCTGCTCCTTCTGCTGCTGCTTCTCCAGCTTTCGCTCCTTCATGCTGCGCAGCTTGGCCTTGCCGATGCCCCCAGCTTGGCGGATGGACTCTAGCAGAGTGGCCAGCCACCGGAGGGGTCAACCACTTCCCTGGGAGCTCCCTGGACTGGAGCCGGGAGGTGGGGAACAGGGCAAGGAGGAAAGGCTGCTCAGGCAGGGCTGGGGAAGCTTACTGTGTCCAAGAGCCTGCTGGGAGGGAAGTCACCTCCCCTCAAACGAGGAGCCCTGCGCTGGGGAGGCCGGACCTTTGGAGACTGTGTGTGGGGGCCTGGGCACTGACTTCTGCAACCACCTGAGCGCGGGCATCCTGTGTGCAGATACTCCCTGCTTCCTCTCTAGCCCCCACCCTGCAGAGCTGGACCCCTGAGCTAGCCATGCTCTGACAGTCTCAGTTGCACACACGAGCCAGCAGAGGGGTTTTGTGCCACTTCTGGATGCTAGGGTTACACTGGGAGACACAGCAGTGAAGCTGAAATGAAAAATGTGTTGCTGTAGTTTGTTATTAGACCCCTTCTTTCCATTGGTTTAATTAGGAATGGGGAACCCAGAGCCTCACTTGTTCAGGCTCCCTCTGCCCTAGAAGTGAGAAGTCCAGAGCTCTACAGTTTGAAAACCACTATTTTATGAACCAAGTAGAACAAGATATTTGAAATGGAAACTATTCAAAAAATTGAGAATTTCTGACCACTTAACAAACCCACAGAAAATCCACCCGAGTGCACTGAGCACGCCAGAAATCAGGTGGCCTCAAAGAGCTGCTCCCACCTGAAGGAGACGCGCTGCTGCTGCTGTCGTCCTGCCTGGCGCCTTGGCCTACAGGGGCCGCGGTTGAGGGTGGGAGTGGGGGTGCACTGGCCAGCACCTCAGGAGCTGGGGGTGGTGGTGGGGGCGGTGGGGGTGGTGTTAGTACCCCATCTTGTAGGTCTGAAACACAAAGTGTGGGGTGTCTAGGGAAGAAGGTGTGTGACCAGGGAGGTCCCCGGCCCAGCTCCCATCCCAGAACCCAGCTCACCTACCTTGAGAGGCTCGGCTACCTCAGTGTGGAAGGTGGGCAGTTCTGGAATGGTGCCAGGGGCAGAGGGGGCAATGCCGGGGCCCAGGTCGGCAATGTACATGAGGTCGTTGGCAATGCCGGGCAGGTCAGGCAGGTAGGATGGAACATCAATCTCAGGCACCTGGCCCAGGTCTGGCACATAGAAGTAGTTCTCTGGGACCTGCAAGATTAGGCAGGGACATGTGAGAGGTGACAGGGACCTGCAGGGGCAGCCAACAAGACCTTGTGTGCACCTCCCATGGGTGGAATAAGGGGCCCAACAGCCTTGACTGGAGAGGAGCTCTGGCAAGGCCCTGGGCCACTGCACCTGTCTCCACCTCTGTCCCACCCCTCCCACCTGCTGTTCCAGCTGCTCTCTCTTGCTGATGGACAAGGGGGCATCAAACAGCTTCTCCTCTGTCTCTGCCCCCAGCATCACATGGGTCTTTGTTACAGCACCAGCCAGGGGGTCCAGGAAGACATACTTCTTCTACCTACAGAGGCGACATGGGGGTCAGGCAAGCTGACACCCGCTGTCCTGAGCCCATGTTCCTCTCCCACATCATCAGGGGCACAGCGTGCACTGTGGGGTCCCAGGCCTCCCGAGCCGAGCCACCCGTCACCCCCTGGCTCCTGGCCTATGTGCTGTACCTGTGTCTGATGCCCTGGGTCCCCACTAAGCCAGGCCGGGCCTCCCGCCCACACCCCTCGGCCCTGCCCTCTGGCCATACAGGTTCTCGGTGGTGTTGAAGAGCAGCAAGGAGCTGACAGAGCTGATGTTGCTGGGAAGACCCCCAAGTCCCTCTTCTGCATCGTCCTCGGGCTCCGGCTTGGTGCTCACGCACACAGGAAAGTCCTTCAGCTTCTCCTGAGAGGGCCAGGATGGCCAAGGGATGGTGAATATTTGGTGCTGGGCCTAATCAGCTGCCATCCCATCCCAGTCAGCCTCCTCTGGGGGACAGAACCCTATGGTGGCCCCGGCTCCTCCCCAGTATCCAGTCCTCCTGGTGTGTGACAGGCTATATGCGCGGCCAGCAGACCTGCAGGGCCCGCTCGTCCAGGGGGCGGTGCTTGCTCTGGATCCTGTGGCGGGGGCGTCTCTGCAGGCCAGGGTCCTGGGCGCCCGTGAAGATGGAGCCATATTCCTGCAGGCGCCCTGGAGCAGGGTACTTGGCACTGGAGAACACCTGTGGACACAGGGACAAGTCTGAGGGGGCCCCAAGAGGCTCAGAGGGCTAGGATTGCTTGGCAGGAGAGGGTGGAGTTGGAAGCCTGGGCGAGAAGAAAGCTCAAGGTACAGGTGGGCAGCAGGGCAGAGACTGGGCA',
 'chromosome': '1',
 'start_pos': 12000,
 'end_pos': 18200}
```

### Data Fields

- `sequence`: a string containing a DNA sequence from the human reference genome
- `chromosome`: a string indicating the chromosome (1,2,...,21,X,Y)
- `start_pos`: an integer indicating the index of the sequence's first nucleotide 
- `end_pos`: an integer indicating the index of the sequence's last nucleotide 

### Data Splits

The Human reference genome dataset has 3 splits: train, validation, and test. Below are the statistics for the dataset.
```
| Dataset Split | Number of Instances in Split (6kb)          |  Number of Instances in Split (12kb)                           |
| ------------- | ------------------------------------------- | -------------------------------------------------------------- |
| Train         | 498,444                                     | 249,222                                                        |
| Validation    | 7,784                                       | 3,892                                                          |
| Test          | 8,469                                       | 4,234                                                          |
```
## Dataset Creation

[N/A]

### Curation Rationale

[N/A]


### Source Data

#### Initial Data Collection and Normalization



The data consists of sequences cut from the chromosomes found in the [GRCh38/hg38](https://www.ncbi.nlm.nih.gov/assembly/GCF_000001405.26) human reference genome.

#### Who are the source language producers?

[N/A]

### Annotations

The dataset does not contain any additional annotations.

#### Annotation process

[N/A]

#### Who are the annotators?

[N/A]

### Personal and Sensitive Information

[N/A]

## Considerations for Using the Data

### Social Impact of Dataset

[N/A]

### Discussion of Biases

[N/A]


### Other Known Limitations

[N/A]


## Additional Information

### Dataset Curators

[N/A]


### Licensing Information

[N/A]

### Citation Information

```bibtex
@article{dalla2023nucleotide,
  title={The Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics},
  author={Dalla-Torre, Hugo and Gonzalez, Liam and Mendoza Revilla, Javier and Lopez Carranza, Nicolas and Henryk Grywaczewski, Adam and Oteri, Francesco and Dallago, Christian and Trop, Evan and Sirelkhatim, Hassan and Richard, Guillaume and others},
  journal={bioRxiv},
  pages={2023--01},
  year={2023},
  publisher={Cold Spring Harbor Laboratory}
}
```
DNA Genomics Nucleotide region:us
{"pretty_name": "Human Reference Genome", "tags": ["DNA", "Genomics", "Nucleotide"]}
TAGS
#DNA #Genomics #Nucleotide #region-us 

# Dataset Card for the human reference genome

## Dataset Description

- Repository: Nucleotide Transformer
- Paper: The Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics

### Dataset Summary

The Human reference genome dataset was constructed by considering all autosomal and sex chromosomes sequences from reference assembly GRCh38/hg38 and reaches a total of 3.2 billion nucleotides.

### Supported Tasks and Leaderboards

This dataset has been used as a pre-training corpus for the Nucleotide Transformers models. Depending on the configuration used, each sequence is 6,200 or 12,200 base pase pairs long. If the dataset is iterated without being shuffled, the first 100 nucleotides of a sequence are the same as the last 100 base pairs of the previous sequence, and the last 100 nucleotides are the same as the first 100 base pairs of the next sequence. During training, this allows for randomly selecting a nucleotide between the first 200 nucleotides of the sequence and start the tokenization from this nucleotide. That way, all the chromosome is covered and the model sees different tokens for a given sequence at each epoch. 

### Languages

DNA

## Dataset Structure

[N/A]


### Data Instances

For each instance, there is a string representing the sequence, a string indicating the chromosome, and two integers representing the index of the first and last nucleotide respectively. An instance is shown below:


### Data Fields

- 'sequence': a string containing a DNA sequence from the human reference genome
- 'chromosome': a string indicating the chromosome (1,2,...,21,X,Y)
- 'start_pos': an integer indicating the index of the sequence's first nucleotide 
- 'end_pos': an integer indicating the index of the sequence's last nucleotide 

### Data Splits

The Human reference genome dataset has 3 splits: train, validation, and test. Below are the statistics for the dataset.

## Dataset Creation

[N/A]

### Curation Rationale

[N/A]


### Source Data

#### Initial Data Collection and Normalization



The data consists of sequences cut from the chromosomes found in the GRCh38/hg38 human reference genome.

#### Who are the source language producers?

[N/A]

### Annotations

The dataset does not contain any additional annotations.

#### Annotation process

[N/A]

#### Who are the annotators?

[N/A]

### Personal and Sensitive Information

[N/A]

## Considerations for Using the Data

### Social Impact of Dataset

[N/A]

### Discussion of Biases

[N/A]


### Other Known Limitations

[N/A]


## Additional Information

### Dataset Curators

[N/A]


### Licensing Information

[N/A]




# Dataset Card for the human reference genome ## Dataset Description

- Repository: Nucleotide Transformer
- Paper: The Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics ### Dataset Summary

The Human reference genome dataset was constructed by considering all autosomal and sex chromosomes sequences from reference assembly GRCh38/hg38 and reaches a total of 3.2 billion nucleotides. ### Supported Tasks and Leaderboards

This dataset has been used as a pre-training corpus for the Nucleotide Transformers models. Depending on the configuration used, each sequence is 6,200 or 12,200 base pase pairs long. If the dataset is iterated without being shuffled, the first 100 nucleotides of a sequence are the same as the last 100 base pairs of the previous sequence, and the last 100 nucleotides are the same as the first 100 base pairs of the next sequence. During training, this allows for randomly selecting a nucleotide between the first 200 nucleotides of the sequence and start the tokenization from this nucleotide. That way, all the chromosome is covered and the model sees different tokens for a given sequence at each epoch. ### Languages

DNA ## Dataset Structure

[N/A] ### Data Instances

For each instance, there is a string representing the sequence, a string indicating the chromosome, and two integers representing the index of the first and last nucleotide respectively. An instance is shown below: ### Data Fields

- 'sequence': a string containing a DNA sequence from the human reference genome
- 'chromosome': a string indicating the chromosome (1,2,...,21,X,Y)
- 'start_pos': an integer indicating the index of the sequence's first nucleotide 
- 'end_pos': an integer indicating the index of the sequence's last nucleotide ### Data Splits

The Human reference genome dataset has 3 splits: train, validation, and test. Below are the statistics for the dataset. ## Dataset Creation

[N/A] ### Curation Rationale

[N/A] ### Source Data #### Initial Data Collection and Normalization



The data consists of sequences cut from the chromosomes found in the GRCh38/hg38 human reference genome. #### Who are the source language producers?

[N/A] ### Annotations

The dataset does not contain any additional annotations. #### Annotation process

[N/A] #### Who are the annotators?

[N/A] ### Personal and Sensitive Information

[N/A] ## Considerations for Using the Data ### Social Impact of Dataset

[N/A] ### Discussion of Biases

[N/A] ### Other Known Limitations

[N/A] ## Additional Information ### Dataset Curators

[N/A] ### Licensing Information

[N/A]
TAGS
#DNA #Genomics #Nucleotide #region-us 
 # Dataset Card for the human reference genome ## Dataset Description

- Repository: Nucleotide Transformer
- Paper: The Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics ### Dataset Summary

The Human reference genome dataset was constructed by considering all autosomal and sex chromosomes sequences from reference assembly GRCh38/hg38 and reaches a total of 3.2 billion nucleotides. ### Supported Tasks and Leaderboards

This dataset has been used as a pre-training corpus for the Nucleotide Transformers models. Depending on the configuration used, each sequence is 6,200 or 12,200 base pase pairs long. If the dataset is iterated without being shuffled, the first 100 nucleotides of a sequence are the same as the last 100 base pairs of the previous sequence, and the last 100 nucleotides are the same as the first 100 base pairs of the next sequence. During training, this allows for randomly selecting a nucleotide between the first 200 nucleotides of the sequence and start the tokenization from this nucleotide. That way, all the chromosome is covered and the model sees different tokens for a given sequence at each epoch. ### Languages

DNA ## Dataset Structure

[N/A] ### Data Instances

For each instance, there is a string representing the sequence, a string indicating the chromosome, and two integers representing the index of the first and last nucleotide respectively. An instance is shown below: ### Data Fields

- 'sequence': a string containing a DNA sequence from the human reference genome
- 'chromosome': a string indicating the chromosome (1,2,...,21,X,Y)
- 'start_pos': an integer indicating the index of the sequence's first nucleotide 
- 'end_pos': an integer indicating the index of the sequence's last nucleotide ### Data Splits

The Human reference genome dataset has 3 splits: train, validation, and test. Below are the statistics for the dataset. ## Dataset Creation

[N/A] ### Curation Rationale

[N/A] ### Source Data #### Initial Data Collection and Normalization



The data consists of sequences cut from the chromosomes found in the GRCh38/hg38 human reference genome. #### Who are the source language producers?

[N/A] ### Annotations

The dataset does not contain any additional annotations. #### Annotation process

[N/A] #### Who are the annotators?

[N/A] ### Personal and Sensitive Information

[N/A] ## Considerations for Using the Data ### Social Impact of Dataset

[N/A] ### Discussion of Biases

[N/A] ### Other Known Limitations

[N/A] ## Additional Information ### Dataset Curators

[N/A] ### Licensing Information

[N/A]