transformers safetensors llama text-generation mergekit merge autotrain_compatible endpoints_compatible text-generation-inference region:us
---
base_model: []
tags:
- mergekit
- merge

---
# TestBunny-7B

This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).

This is mostly me experimenting with blending different models together in various ways. 
With the adjusted config file, it writes coherently out to at least 16k and potentially up to the full 32k (untested). 

## Merge Details
### Merge Method

This model was merged using a combination of TIES and task-arithmetic via mergekit-mega megamerge. 

### Models Merged

The following models were included in the merge:
* teknium/OpenHermes-2.5-Mistral-7B
* Open-Orca/Mistral-7B-OpenOrca
* cognitivecomputations/dolphin-2.6-mistral-7b
* NousResearch/Nous-Capybara-7B-V1.9
* lemonilia/LimaRP-Mistral-7B-v0.1

### Configuration

The following YAML configuration was used to produce this model:

```yaml
# Base Model
models:
  - model: teknium/OpenHermes-2.5-Mistral-7B
    parameters:
      density: 0.5 
      weight: 0.5
  - model: Open-Orca/Mistral-7B-OpenOrca
    parameters:
      density: 0.6
      weight: 1.0 
  - model: cognitivecomputations/dolphin-2.6-mistral-7b
    parameters:
      density: 0.6
      weight: 0.7
  - model: lemonilia/LimaRP-Mistral-7B-v0.1
    parameters:
      density: 0.5
      weight: 0.2
merge_method: ties
base_model: mistralai/Mistral-7B-Instruct-v0.2
parameters:
  normalize: true
  int8_mask: true
dtype: float16
name: cetherald-base
---
# Tone it down a little
models:
  - model: cetherald-base
    parameters:
      density: 1.0 
      weight: 1.0
  - model: NousResearch/Nous-Capybara-7B-V1.9
    parameters:
      density: 0.2
      weight: 0.15
  - model: lemonilia/LimaRP-Mistral-7B-v0.1
    parameters:
      density: 0.2
      weight: 0.08 
merge_method: ties
base_model: mistralai/Mistral-7B-v0.1
parameters:
  normalize: true
  int8_mask: true
dtype: float16
name: bunny-a
---
# Add just a little more RP style
models:
  - model: bunny-a
    parameters:
      weight: 1.0
  - model: lemonilia/LimaRP-Mistral-7B-v0.1
    parameters:
      weight: 0.08 
merge_method: task_arithmetic
base_model: mistralai/Mistral-7B-v0.1
dtype: float16
```
