transformers safetensors mistral text-generation mergekit merge arxiv:2212.04089 base_model:sethuiyer/Dr_Samantha_7b_mistral base_model:teknium/Mistral-Trismegistus-7B autotrain_compatible endpoints_compatible text-generation-inference region:us
---
base_model:
- sethuiyer/Dr_Samantha_7b_mistral
- teknium/Mistral-Trismegistus-7B
tags:
- mergekit
- merge

---
# labrabbit7b

This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).

Another (failed) experiment of mine. The writing style on this model is good, but it suffers from severe wrap-up bias especially in storywriting applications. 

## Merge Details
### Merge Method

This model was merged using SLERP and the [task arithmetic](https://arxiv.org/abs/2212.04089) merge method using mistralai/Mistral-7B-v0.1 as a base.

### Models Merged

The following models were included in the merge:
* [sethuiyer/Dr_Samantha_7b_mistral](https://huggingface.co/sethuiyer/Dr_Samantha_7b_mistral)
* [teknium/Mistral-Trismegistus-7B](https://huggingface.co/teknium/Mistral-Trismegistus-7B)
* [cognitivecomputations/dolphin-2.6-mistral-7b](https://huggingface.co/cognitivecomputations/dolphin-2.6-mistral-7b)
* [Open-Orca/Mistral-7B-OpenOrca](https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca)
* [lemonilia/LimaRP-Mistral-7B-v0.1](https://huggingface.co/lemonilia/LimaRP-Mistral-7B-v0.1)

### Configuration

The following YAML configuration was used to produce this model:

```yaml
models:
  - model: cognitivecomputations/dolphin-2.6-mistral-7b
  - model: Open-Orca/Mistral-7B-OpenOrca
merge_method: slerp
base_model: cognitivecomputations/dolphin-2.6-mistral-7b
parameters:
  t:
    - filter: self_attn
      value: [0, 0.5, 0.3, 0.7, 1]
    - filter: mlp
      value: [1, 0.5, 0.7, 0.3, 0]
    - value: 0.5 # fallback for rest of tensors
dtype: float16
name: cetacean-slerp-7b
---
base_model: mistralai/Mistral-7B-v0.1
dtype: float16
merge_method: task_arithmetic
slices:
- sources:
  - layer_range: [0, 32]
    model: mistralai/Mistral-7B-v0.1
  - layer_range: [0, 32]
    model: cetacean-slerp-7b
    parameters:
      weight: 1.0
  - layer_range: [0, 32]
    model: sethuiyer/Dr_Samantha_7b_mistral
    parameters:
      weight: 0.2
  - layer_range: [0, 32]
    model: teknium/Mistral-Trismegistus-7B
    parameters:
      weight: 0.15
  - layer_range: [0, 32]
    model: lemonilia/LimaRP-Mistral-7B-v0.1
    parameters:
      weight: 0.1
```
