# BigAurelian v0.5 120b 32k

<img src="https://cdn-uploads.huggingface.co/production/uploads/65a6db055c58475cf9e6def1/QqGLOoVkdN-Z4kLD6OWP6.png" width=600>

A Goliath-120b style frankenmerge of aurelian-v0.5-70b-32K and WinterGoddess-1.4x-70b. The goal is to have similar performance with an extended context size. **Important:** Use a positional embeddings compression factor (**compress_pos_emb**) of **8** when loading this model.

# Prompting Format
Llama2 and Alpaca.

# Merge process
The models used in the merge are [aurelian-v0.5-70b-32K](https://huggingface.co/grimulkan/aurelian-v0.5-70b-rope8-32K-fp16) and [WinterGoddess-1.4x-70b](https://huggingface.co/Sao10K/WinterGoddess-1.4x-70B-L2).

The layer mix:
```yaml
- range 0, 16
  aurelian
- range 8, 24
  WinterGoddess
- range 17, 32
  aurelian
- range 25, 40
  WinterGoddess
- range 33, 48
  aurelian
- range 41, 56
  WinterGoddess
- range 49, 64
  aurelian
- range 57, 72
  WinterGoddess
- range 65, 80
  aurelian
```

# Acknowledgements
[@grimulkan](https://huggingface.co/grimulkan) For creating aurelian-v0.5-70b-32K

[@Sao10K](https://huggingface.co/Sao10K) For creating WinterGoddess

[@alpindale](https://huggingface.co/alpindale) For creating the original Goliath

[@chargoddard](https://huggingface.co/chargoddard) For developing [mergekit](https://github.com/cg123/mergekit).

{"language": ["en"], "license": "llama2", "tags": ["aurelian", "WinterGoddess", "frankenmerge", "120b", "32k"], "pipeline_tag": "conversational"}
transformers safetensors llama text-generation aurelian WinterGoddess frankenmerge 120b 32k conversational en license:llama2 autotrain_compatible endpoints_compatible text-generation-inference region:us