region:us
# Processed whole-genome alignment of 100 vertebrate species
For more information check out our [paper](https://doi.org/10.1101/2023.10.10.561776) and [repository](https://github.com/songlab-cal/gpn).

Source data:
- MSA was downloaded from http://hgdownload.soe.ucsc.edu/goldenPath/hg38/multiz100way/
- Human sequence was replaced with a newer reference:
  http://ftp.ensembl.org/pub/release-107/fasta/homo_sapiens/dna/Homo_sapiens.GRCh38.dna_sm.primary_assembly.fa.gz

Available MSAs:
- `89.zarr.zip` contains human + 89 vertebrates (excluding 10 closest primates)
- `99.zarr.zip` contains human + 99 vertebrates

Example usage:
```python
from gpn.data import GenomeMSA

genome_msa = GenomeMSA(msa_path)
X = genome_msa.get_msa(chrom, start, end, strand="+", tokenize=False)
```

Coordinates:
- `hg38` assembly
- `chrom` should be in `["1", "2", ..., "22", "X", "Y"]`

## Streaming (playing, few VEP queries)
- Faster setup (no need to download and unzip)
- Slower queries (depends on network connection)
- Multiple dataloader workers don't seem to work
- More CPU memory required to load: 10.41 GB 
- Recommended if you just want to do a few queries, e.g. VEP for a couple thousand variants
- ```python
  msa_path = "zip:///::https://huggingface.co/datasets/songlab/multiz100way/resolve/main/89.zarr.zip"
  ```

## Local download (training, large-scale VEP)
- Requires downloading (34GB) and unzipping (currently quite slow, will try to improve)
  ```bash
  wget https://huggingface.co/datasets/songlab/multiz100way/resolve/main/89.zarr.zip
  7z x 89.zarr.zip -o89.zarr  # can still take 5 hours with 32 cores, will try to streamline this in the future
  ```
- Much faster to query
- Can have multiple dataloader workers
- Virtually no CPU memory required to load
- Recommended for training or VEP for millions of variants
- ```python
  msa_path = "89.zarr"
  ```