
<p align="center">
  <img src="https://codeberg.org/aninokuma/DeydooAssistant/raw/branch/main/logo.webp" height="256px" alt="SynthIQ">
</p>

# SynthIQ

This is SynthIQ, rated **92.23/100** by GPT-4 across varied complex prompts. I used [mergekit](https://github.com/cg123/mergekit) to merge models.

| Benchmark Name   | Score |
| ---------------- | ----- |
| ARC              | 65.87 |
| HellaSwag        | 85.82 |
| MMLU             | 64.75 |
| TruthfulQA       | 57.00 |
| Winogrande       | 78.69 |
| GSM8K            | 64.06 |
| AGIEval          | 42.67 |
| GPT4All          | 73.71 |
| Bigbench         | 44.59 |


## Update - 19/01/2024
Tested to work well with autogen and CrewAI

GGUF Files

[Q4_K_M](https://huggingface.co/sethuiyer/SynthIQ_GGUF/blob/main/synthiq.Q4_K_M.gguf) - medium, balanced quality - recommended

[Q_6_K](https://huggingface.co/sethuiyer/SynthIQ_GGUF/blob/main/synthiq.Q6_K.gguf) - very large, extremely low quality loss

[Q8_0](https://huggingface.co/sethuiyer/SynthIQ_GGUF/blob/main/synthiq.Q8.gguf) - very large, extremely low quality loss - not recommended

**Important Update**: SynthIQ is now available on Ollama. You can use it by running the command ```ollama run stuehieyr/synthiq``` in your 
terminal. If you have limited computing resources, check out this [video](https://www.youtube.com/watch?v=Qa1h7ygwQq8) to learn how to run it on 
a Google Colab backend.


# Yaml Config

```yaml

slices:
  - sources:
      - model: Weyaxi/OpenHermes-2.5-neural-chat-v3-3-openchat-3.5-1210-Slerp
        layer_range: [0, 32]
      - model: uukuguy/speechless-mistral-six-in-one-7b
        layer_range: [0, 32]

merge_method: slerp
base_model: mistralai/Mistral-7B-v0.1

parameters:
  t:
    - filter: self_attn
      value: [0, 0.5, 0.3, 0.7, 1]
    - filter: mlp
      value: [1, 0.5, 0.7, 0.3, 0]
    - value: 0.5 # fallback for rest of tensors
tokenizer_source: union

dtype: bfloat16

```

<!-- prompt-template start -->
## Prompt template: ChatML

```
<|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

```

<!-- prompt-template end -->


License is LLama2 license as uukuguy/speechless-mistral-six-in-one-7b is llama2 license.

# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_sethuiyer__SynthIQ-7b)

# [Nous Benchmark Evalation Results](https://huggingface.co/spaces/mlabonne/Yet_Another_LLM_Leaderboard)
Detailed results can be found [here](https://gist.github.com/sethuiyer/f47dee388a4e95d46181c98d37d66a58)
{"language": ["en"], "license": "llama2", "library_name": "transformers", "tags": ["mistral", "merge"], "datasets": ["stingning/ultrachat", "garage-bAInd/Open-Platypus", "Open-Orca/OpenOrca", "TIGER-Lab/MathInstruct", "OpenAssistant/oasst_top1_2023-08-25", "teknium/openhermes", "meta-math/MetaMathQA", "Open-Orca/SlimOrca"], "pipeline_tag": "text-generation", "base_model": ["Weyaxi/OpenHermes-2.5-neural-chat-v3-3-openchat-3.5-1210-Slerp", "ehartford/dolphin-2.1-mistral-7b", "Open-Orca/Mistral-7B-OpenOrca", "bhenrym14/mistral-7b-platypus-fp16", "ehartford/samantha-1.2-mistral-7b", "iteknium/CollectiveCognition-v1.1-Mistral-7B", "HuggingFaceH4/zephyr-7b-alpha"], "model-index": [{"name": "sethuiyer/SynthIQ-7b", "results": [{"task": {"type": "text-generation", "name": "Text Generation"}, "dataset": {"name": "AI2 Reasoning Challenge (25-Shot)", "type": "ai2_arc", "config": "ARC-Challenge", "split": "test", "args": {"num_few_shot": 25}}, "metrics": [{"type": "acc_norm", "value": 65.87, "name": "normalized accuracy"}], "source": {"url": "https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/SynthIQ-7b", "name": "Open LLM Leaderboard"}}, {"task": {"type": "text-generation", "name": "Text Generation"}, "dataset": {"name": "HellaSwag (10-Shot)", "type": "hellaswag", "split": "validation", "args": {"num_few_shot": 10}}, "metrics": [{"type": "acc_norm", "value": 85.82, "name": "normalized accuracy"}], "source": {"url": "https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/SynthIQ-7b", "name": "Open LLM Leaderboard"}}, {"task": {"type": "text-generation", "name": "Text Generation"}, "dataset": {"name": "MMLU (5-Shot)", "type": "cais/mmlu", "config": "all", "split": "test", "args": {"num_few_shot": 5}}, "metrics": [{"type": "acc", "value": 64.75, "name": "accuracy"}], "source": {"url": "https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/SynthIQ-7b", "name": "Open LLM Leaderboard"}}, {"task": {"type": "text-generation", "name": "Text Generation"}, "dataset": {"name": "TruthfulQA (0-shot)", "type": "truthful_qa", "config": "multiple_choice", "split": "validation", "args": {"num_few_shot": 0}}, "metrics": [{"type": "mc2", "value": 57}], "source": {"url": "https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/SynthIQ-7b", "name": "Open LLM Leaderboard"}}, {"task": {"type": "text-generation", "name": "Text Generation"}, "dataset": {"name": "Winogrande (5-shot)", "type": "winogrande", "config": "winogrande_xl", "split": "validation", "args": {"num_few_shot": 5}}, "metrics": [{"type": "acc", "value": 78.69, "name": "accuracy"}], "source": {"url": "https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/SynthIQ-7b", "name": "Open LLM Leaderboard"}}, {"task": {"type": "text-generation", "name": "Text Generation"}, "dataset": {"name": "GSM8k (5-shot)", "type": "gsm8k", "config": "main", "split": "test", "args": {"num_few_shot": 5}}, "metrics": [{"type": "acc", "value": 64.06, "name": "accuracy"}], "source": {"url": "https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/SynthIQ-7b", "name": "Open LLM Leaderboard"}}]}]}
transformers safetensors mistral text-generation merge conversational en dataset:stingning/ultrachat dataset:garage-bAInd/Open-Platypus dataset:Open-Orca/OpenOrca dataset:TIGER-Lab/MathInstruct dataset:OpenAssistant/oasst_top1_2023-08-25 dataset:teknium/openhermes dataset:meta-math/MetaMathQA dataset:Open-Orca/SlimOrca base_model:Weyaxi/OpenHermes-2.5-neural-chat-v3-3-openchat-3.5-1210-Slerp base_model:ehartford/dolphin-2.1-mistral-7b base_model:Open-Orca/Mistral-7B-OpenOrca base_model:bhenrym14/mistral-7b-platypus-fp16 base_model:ehartford/samantha-1.2-mistral-7b base_model:iteknium/CollectiveCognition-v1.1-Mistral-7B base_model:HuggingFaceH4/zephyr-7b-alpha license:llama2 model-index autotrain_compatible endpoints_compatible text-generation-inference region:us