text
🔁 AutoMerger created the best 7B model on the Open LLM Leaderboard
🔁 AutoMerger created the best 7B model on the Open LLM Leaderboard
new_line
new_line
text
By randomly combining top models from the Open LLM Leaderboard, AutoMerger created YamshadowExperiment28-7B. The model is three weeks old and has been at the top of the leaderboard for a week now. It was created through a simple SLERP merge of:
By randomly combining top models from the Open LLM Leaderboard, AutoMerger created YamshadowExperiment28-7B. The model is three weeks old and has been at the top of the leaderboard for a week now. It was created through a simple SLERP merge of:
new_line
new_line
text
- 
- 
resource
https://huggingface.co/automerger/YamShadow-7B
model
automerger/YamShadow-7B
https://huggingface.co/automerger/YamShadow-7B
text
 (another top model created by AutoMerger)
 (another top model created by AutoMerger)
new_line
text
- 
- 
resource
https://huggingface.co/yam-peleg/Experiment28-7B
model
yam-peleg/Experiment28-7B
https://huggingface.co/yam-peleg/Experiment28-7B
text
 (a top model from 
 (a top model from 
mention
@yam-peleg
yam-peleg
text
)
)
new_line
new_line
text
1/ On the Open LLM Leaderboard, it managed to outperform the excellent M7-7b model, which has been the #1 7B model for a while now.
1/ On the Open LLM Leaderboard, it managed to outperform the excellent M7-7b model, which has been the #1 7B model for a while now.
new_line
new_line
text
2/ On the YALL leaderboard, YamshadowExperiment28-7B is ranked as the 9th best-performing automerge (but note that the scores are very close to each other). Compared to others, it does not perform particularly well on AGIEval or Bigbench.
2/ On the YALL leaderboard, YamshadowExperiment28-7B is ranked as the 9th best-performing automerge (but note that the scores are very close to each other). Compared to others, it does not perform particularly well on AGIEval or Bigbench.
new_line
new_line
text
3/ Thanks to 
3/ Thanks to 
mention
@sam-paech
sam-paech
text
 , I have scores on EQ-Bench, where it managed to outperform all of my previous models. It even surpasses recent models such as DBRX instruct, Qwen1.5 32B Chat, and Cohere's Command R+.
 , I have scores on EQ-Bench, where it managed to outperform all of my previous models. It even surpasses recent models such as DBRX instruct, Qwen1.5 32B Chat, and Cohere's Command R+.
new_line
new_line
text
Surprisingly, it does not support ChatML or Mistral Instruct, unlike my other merges (which are part of its family tree). Alpaca works well 99% of the time, but the model can sometimes produce a lot of "INST" tokens for no reason.
Surprisingly, it does not support ChatML or Mistral Instruct, unlike my other merges (which are part of its family tree). Alpaca works well 99% of the time, but the model can sometimes produce a lot of "INST" tokens for no reason.
new_line
new_line
text
In my experiments, YamshadowExperiment28-7B doesn't seem smarter than other successful merges like AlphaMonarch. On the contrary, I found several mathematical or reasoning problems where it fails.
In my experiments, YamshadowExperiment28-7B doesn't seem smarter than other successful merges like AlphaMonarch. On the contrary, I found several mathematical or reasoning problems where it fails.
new_line
new_line
text
Considering these results, it looks like it might overfit the Open LLM Leaderboard. I guess it's anything but surprising when you randomly merge 156 models.
Considering these results, it looks like it might overfit the Open LLM Leaderboard. I guess it's anything but surprising when you randomly merge 156 models.
new_line
new_line
text
🤗 Model: 
🤗 Model: 
resource
https://huggingface.co/automerger/YamshadowExperiment28-7B
model
automerger/YamshadowExperiment28-7B
https://huggingface.co/automerger/YamshadowExperiment28-7B
new_line
text
🔁 AutoMerger: 
🔁 AutoMerger: 
resource
https://huggingface.co/spaces/mlabonne/AutoMerger
space
mlabonne/AutoMerger
https://huggingface.co/spaces/mlabonne/AutoMerger
🔁 AutoMerger created the best 7B model on the Open LLM Leaderboard

By randomly combining top models from the Open LLM Leaderboard, AutoMerger created YamshadowExperiment28-7B. The model is three weeks old and has been at the top of the leaderboard for a week now. It was created through a simple SLERP merge of:

- https://huggingface.co/automerger/YamShadow-7B (another top model created by AutoMerger)
- https://huggingface.co/yam-peleg/Experiment28-7B (a top model from @yam-peleg)

1/ On the Open LLM Leaderboard, it managed to outperform the excellent M7-7b model, which has been the #1 7B model for a while now.

2/ On the YALL leaderboard, YamshadowExperiment28-7B is ranked as the 9th best-performing automerge (but note that the scores are very close to each other). Compared to others, it does not perform particularly well on AGIEval or Bigbench.

3/ Thanks to @sam-paech , I have scores on EQ-Bench, where it managed to outperform all of my previous models. It even surpasses recent models such as DBRX instruct, Qwen1.5 32B Chat, and Cohere's Command R+.

Surprisingly, it does not support ChatML or Mistral Instruct, unlike my other merges (which are part of its family tree). Alpaca works well 99% of the time, but the model can sometimes produce a lot of "INST" tokens for no reason.

In my experiments, YamshadowExperiment28-7B doesn't seem smarter than other successful merges like AlphaMonarch. On the contrary, I found several mathematical or reasoning problems where it fails.

Considering these results, it looks like it might overfit the Open LLM Leaderboard. I guess it's anything but surprising when you randomly merge 156 models.

🤗 Model: https://huggingface.co/automerger/YamshadowExperiment28-7B
🔁 AutoMerger: https://huggingface.co/spaces/mlabonne/AutoMerger
🔥
victor AtAndDev Tonic vladbogo rmdhirr samusenps Gargaz osanseviero kgourgou Kukedlc Ramikan-BR
👍
neeravkumar Heng666 Gargaz osanseviero mylesgoose kgourgou eliolio Muhammad2003 Ramikan-BR
🚀
Ramikan-BR
👀
Ramikan-BR
❤️
Ramikan-BR