
# Medorca-7B-Ties

Medorca-7B-Ties is a merge of the following models:
* [microsoft/Orca-2-7b](https://huggingface.co/microsoft/Orca-2-7b)
* [epfl-llm/meditron-7b](https://huggingface.co/epfl-llm/meditron-7b)

## 🧩 Configuration

```yaml
models:
  - model: NousResearch/Llama-2-7b-hf
    # no parameters necessary for base model
  - model: microsoft/Orca-2-7b
    parameters:
      density: 0.5
      weight: 0.3
  - model: epfl-llm/meditron-7b
    parameters:
      density: 0.5
      weight: 0.5
merge_method: ties
base_model: NousResearch/Llama-2-7b-hf
parameters:
  normalize: true
  int8_mask: true
dtype: float16
```

## 💻 Usage

```python
!pip install -qU transformers accelerate

from transformers import AutoTokenizer
import transformers
import torch

model = "Technoculture/Medorca-7B-Ties"
messages = [{"role": "user", "content": "I am feeling sleepy these days"}]

tokenizer = AutoTokenizer.from_pretrained(model)
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    torch_dtype=torch.float16,
    device_map="auto",
)

outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])
```

# Evals

Using [llm-autoeval](https://github.com/Technoculture/llm-autoeval)

![image/png](https://cdn-uploads.huggingface.co/production/uploads/63486df1f8f01fcc4b23e97d/fCILSu-2d1UfY3d0-MMpG.png)

|                                Model                                | ARC |HellaSwag|           MMLU           |TruthfulQA|Winogrande|GSM8K|
|---------------------------------------------------------------------|----:|--------:|--------------------------|---------:|---------:|----:|
|[Medorca-7B-Ties](https://huggingface.co/Technoculture/Medorca-7B-Ties)|47.27|     72.1|Error: File does not exist|     41.52|     70.01| 9.86|

### ARC
|    Task     |Version|       Metric       |    Value    |   |Stderr|
|-------------|-------|--------------------|-------------|---|------|
|arc_challenge|Yaml   |acc,none            |         0.44|   |      |
|             |       |acc_stderr,none     |         0.01|   |      |
|             |       |acc_norm,none       |         0.47|   |      |
|             |       |acc_norm_stderr,none|         0.01|   |      |
|             |       |alias               |arc_challenge|   |      |

Average: 47.27%

### HellaSwag
|  Task   |Version|       Metric       |  Value  |   |Stderr|
|---------|-------|--------------------|---------|---|------|
|hellaswag|Yaml   |acc,none            |     0.54|   |      |
|         |       |acc_stderr,none     |        0|   |      |
|         |       |acc_norm,none       |     0.72|   |      |
|         |       |acc_norm_stderr,none|        0|   |      |
|         |       |alias               |hellaswag|   |      |

Average: 72.1%

### MMLU

Average: Error: File does not exist%

### TruthfulQA
|     Task     |Version|        Metric         |      Value      |   |Stderr|
|--------------|-------|-----------------------|-----------------|---|------|
|truthfulqa    |N/A    |bleu_max,none          |            17.68|   |      |
|              |       |bleu_max_stderr,none   |             0.51|   |      |
|              |       |bleu_acc,none          |             0.42|   |      |
|              |       |bleu_acc_stderr,none   |                0|   |      |
|              |       |bleu_diff,none         |             1.44|   |      |
|              |       |bleu_diff_stderr,none  |             0.46|   |      |
|              |       |rouge1_max,none        |            38.26|   |      |
|              |       |rouge1_max_stderr,none |             0.91|   |      |
|              |       |rouge1_acc,none        |             0.43|   |      |
|              |       |rouge1_acc_stderr,none |                0|   |      |
|              |       |rouge1_diff,none       |             2.39|   |      |
|              |       |rouge1_diff_stderr,none|             0.86|   |      |
|              |       |rouge2_max,none        |            25.29|   |      |
|              |       |rouge2_max_stderr,none |             0.95|   |      |
|              |       |rouge2_acc,none        |             0.35|   |      |
|              |       |rouge2_acc_stderr,none |                0|   |      |
|              |       |rouge2_diff,none       |             2.25|   |      |
|              |       |rouge2_diff_stderr,none|                1|   |      |
|              |       |rougeL_max,none        |            35.85|   |      |
|              |       |rougeL_max_stderr,none |             0.89|   |      |
|              |       |rougeL_acc,none        |             0.42|   |      |
|              |       |rougeL_acc_stderr,none |                0|   |      |
|              |       |rougeL_diff,none       |             2.44|   |      |
|              |       |rougeL_diff_stderr,none|             0.87|   |      |
|              |       |acc,none               |             0.32|   |      |
|              |       |acc_stderr,none        |             0.05|   |      |
|              |       |alias                  |truthfulqa       |   |      |
|truthfulqa_gen|Yaml   |bleu_max,none          |            17.68|   |      |
|              |       |bleu_max_stderr,none   |             0.71|   |      |
|              |       |bleu_acc,none          |             0.42|   |      |
|              |       |bleu_acc_stderr,none   |             0.02|   |      |
|              |       |bleu_diff,none         |             1.44|   |      |
|              |       |bleu_diff_stderr,none  |             0.68|   |      |
|              |       |rouge1_max,none        |            38.26|   |      |
|              |       |rouge1_max_stderr,none |             0.95|   |      |
|              |       |rouge1_acc,none        |             0.43|   |      |
|              |       |rouge1_acc_stderr,none |             0.02|   |      |
|              |       |rouge1_diff,none       |             2.39|   |      |
|              |       |rouge1_diff_stderr,none|             0.93|   |      |
|              |       |rouge2_max,none        |            25.29|   |      |
|              |       |rouge2_max_stderr,none |             0.97|   |      |
|              |       |rouge2_acc,none        |             0.35|   |      |
|              |       |rouge2_acc_stderr,none |             0.02|   |      |
|              |       |rouge2_diff,none       |             2.25|   |      |
|              |       |rouge2_diff_stderr,none|                1|   |      |
|              |       |rougeL_max,none        |            35.85|   |      |
|              |       |rougeL_max_stderr,none |             0.94|   |      |
|              |       |rougeL_acc,none        |             0.42|   |      |
|              |       |rougeL_acc_stderr,none |             0.02|   |      |
|              |       |rougeL_diff,none       |             2.44|   |      |
|              |       |rougeL_diff_stderr,none|             0.93|   |      |
|              |       |alias                  | - truthfulqa_gen|   |      |
|truthfulqa_mc1|Yaml   |acc,none               |             0.27|   |      |
|              |       |acc_stderr,none        |             0.02|   |      |
|              |       |alias                  | - truthfulqa_mc1|   |      |
|truthfulqa_mc2|Yaml   |acc,none               |             0.42|   |      |
|              |       |acc_stderr,none        |             0.01|   |      |
|              |       |alias                  | - truthfulqa_mc2|   |      |

Average: 41.52%

### Winogrande
|   Task   |Version|    Metric     |  Value   |   |Stderr|
|----------|-------|---------------|----------|---|------|
|winogrande|Yaml   |acc,none       |      0.70|   |      |
|          |       |acc_stderr,none|      0.01|   |      |
|          |       |alias          |winogrande|   |      |

Average: 70.01%

### GSM8K
|Task |Version|           Metric            |Value|   |Stderr|
|-----|-------|-----------------------------|-----|---|------|
|gsm8k|Yaml   |exact_match,get-answer       | 0.10|   |      |
|     |       |exact_match_stderr,get-answer| 0.01|   |      |
|     |       |alias                        |gsm8k|   |      |

Average: 9.86%

Average score: Not available due to errors
Elapsed time: 03:48:33
{"license": "apache-2.0", "tags": ["merge", "mergekit", "microsoft/Orca-2-7b", "epfl-llm/meditron-7b"]}
transformers safetensors llama text-generation merge mergekit microsoft/Orca-2-7b epfl-llm/meditron-7b license:apache-2.0 autotrain_compatible endpoints_compatible text-generation-inference region:us