
![image/png](https://i.ibb.co/VYkXDnn/icon.png)

Test merge, attempt to gain the roleplaying capabilities of Undi95/Toppy-M-7B and SanjiWatsuki/Loyal-Macaroni-Maid-7B while maintaining the context and capabilities of the original mistralai/Mistral-7B-Instruct-v0.2

The idea was that by combining two models with one self-merge, it would be possible to make each layer more unique, and therefore make the model “smarter” than a regular self-merge.

### 10.7B Loyal Mistral Maid v0.2
```
slices:
  - sources:
      - model: Mistral_Instruct_SelfMerge
        layer_range: [0, 48]
      - model: Loyal_Toppy_Maid
        layer_range: [0, 48]
merge_method: slerp
base_model: Mistral_Instruct_SelfMerge
parameters:
  t:
    - filter: self_attn
      value: [0, 0.5, 0.3, 0.7, 1]
    - filter: mlp
      value: [1, 0.5, 0.7, 0.3, 0]
    - value: 0.5 # fallback for rest of tensors
dtype: bfloat16
```

### Loyal Toppy Maid
```
slices:
  - sources:
    - model: Undi95/Toppy-M-7B
      layer_range: [0, 24]
  - sources:
    - model: SanjiWatsuki/Loyal-Macaroni-Maid-7B
      layer_range: [8, 32]
merge_method: passthrough
dtype: bfloat16
```

### Mistral_Instruct_SelfMerge
```
slices:
  - sources:
    - model: mistralai/Mistral-7B-Instruct-v0.2
      layer_range: [0, 24]
  - sources:
    - model: mistralai/Mistral-7B-Instruct-v0.2
      layer_range: [8, 32]
merge_method: passthrough
dtype: bfloat16
```
{"language": ["en"], "license": "apache-2.0", "tags": ["merge"]}
transformers safetensors mistral text-generation merge conversational en license:apache-2.0 autotrain_compatible endpoints_compatible text-generation-inference region:us