--- base_model: - mistralai/Mistral-7B-v0.1 - argilla/distilabeled-OpenHermes-2.5-Mistral-7B - NeverSleep/Noromaid-7B-0.4-DPO - senseable/WestLake-7B-v2 - mlabonne/AlphaMonarch-7B library_name: transformers tags: - mergekit - merge license: cc-by-nc-4.0 --- # WestMaid_HermesMonarchv0.1 drawing This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit) ## Merge Details ### Merge Method This model was merged using the [DARE](https://arxiv.org/abs/2311.03099) [TIES](https://arxiv.org/abs/2306.01708) merge method using [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) as a base. ### Models Merged The following models were included in the merge: * [mlabonne/AlphaMonarch-7B](https://huggingface.co/mlabonne/AlphaMonarch-7B) * [NeverSleep/Noromaid-7B-0.4-DPO](https://huggingface.co/NeverSleep/Noromaid-7B-0.4-DPO) * [senseable/WestLake-7B-v2](https://huggingface.co/senseable/WestLake-7B-v2) * [argilla/distilabeled-OpenHermes-2.5-Mistral-7B](https://huggingface.co/argilla/distilabeled-OpenHermes-2.5-Mistral-7B) ### Configuration The following YAML configuration was used to produce this model: ```yaml models: - model: mistralai/Mistral-7B-v0.1 # No parameters necessary for base model - model: senseable/WestLake-7B-v2 parameters: density: 0.58 weight: [0.50, 0.40, 0.25, 0.05] - model: NeverSleep/Noromaid-7B-0.4-DPO parameters: density: 0.58 weight: [0.05, 0.05, 0.25, 0.40] - model: argilla/distilabeled-OpenHermes-2.5-Mistral-7B parameters: density: 0.58 weight: [0.40, 0.50, 0.25, 0.05] - model: mlabonne/AlphaMonarch-7B parameters: density: 0.58 weight: [0.05, 0.05, 0.25, 0.50] merge_method: dare_ties base_model: mistralai/Mistral-7B-v0.1 parameters: int8_mask: true dtype: bfloat16 ``` ## Benchmark Testing ### MT-Bench ![image/png](https://cdn-uploads.huggingface.co/production/uploads/655a9883cbbaec115c3fd6b3/H2BLoovTbLg8d8mtFSKYB.png) ### EQ-Bench Leaderboard drawing ### Table of Benchmarks | | MT-Bench | EQ-Bench v2.1 | |---------------------------------------------------------|---------------------------------------------|---------------------------------------------| | giraffe176/WestLake_Noromaid_OpenHermes_neural-chatv0.1 | **8.021875** | **77.41** (1 Shot, ooba) | | claude-v1 | 7.900000 | 76.83 | | AlphaMonarch-7B | 7.928125 | 76.08 | | gpt-3.5-turbo | 7.943750 | 71.74 | | | [(Paper)](https://arxiv.org/abs/2306.05685) | [(Paper)](https://arxiv.org/abs/2312.06281) [Leaderboard](https://eqbench.com/) |