text
I finished my model merging experiment day.🤗I would love your thoughts on this.
I finished my model merging experiment day.🤗I would love your thoughts on this.
new_line
new_line
text
What did I do? I merged Mistral Instruct 0.1 and 0.2 models using different merging techniques:
What did I do? I merged Mistral Instruct 0.1 and 0.2 models using different merging techniques:
new_line
text
- SLERP: linear interpolation (most popular method)
- SLERP: linear interpolation (most popular method)
new_line
text
- MoE: replace some forward layers with MoE layers; using a random gate for now
- MoE: replace some forward layers with MoE layers; using a random gate for now
new_line
text
- Frankenmerge: also known as passthrough, but that isn't very cool. It concatenates some specified layers ending in different numbers of params. In my case, I went from 7B to 9B.
- Frankenmerge: also known as passthrough, but that isn't very cool. It concatenates some specified layers ending in different numbers of params. In my case, I went from 7B to 9B.
new_line
new_line
text
Note: merging is not building an ensemble of models. You can read more about merging techniques at 
Note: merging is not building an ensemble of models. You can read more about merging techniques at 
link
https://huggingface.co/blog/mlabonne/merge-models
https://huggingface.co/blog/mlabonne/merge-models
new_line
new_line
text
Results
Results
new_line
text
I built the 3 models using mergekit (running in an HF Space) - took less than an hour to do the three) 
I built the 3 models using mergekit (running in an HF Space) - took less than an hour to do the three) 
resource
https://huggingface.co/collections/osanseviero/mistral-instruct-merges-659ebf35ca0781acdb86bb0a
collection
osanseviero/mistral-instruct-merges-659ebf35ca0781acdb86bb0a
https://huggingface.co/collections/osanseviero/mistral-instruct-merges-659ebf35ca0781acdb86bb0a
new_line
new_line
text
I'm doing a quick check with the OpenLLM Leaderboard. 
I'm doing a quick check with the OpenLLM Leaderboard. 
new_line
text
🚨The OpenLLM Leaderboard is more suitable for pre-trained models than instruct models, but I still thought it would be interesting to look at the insights🚨
🚨The OpenLLM Leaderboard is more suitable for pre-trained models than instruct models, but I still thought it would be interesting to look at the insights🚨
new_line
new_line
text
You can look at the attached image. Some interesting things
You can look at the attached image. Some interesting things
new_line
text
- All three models performed somewhere between 0.1 and 0.2 - congrats to the 140 people who got it right in 
- All three models performed somewhere between 0.1 and 0.2 - congrats to the 140 people who got it right in 
link
https://twitter.com/osanseviero/status/1745071548866736171
https://twitter.com/osanseviero/status/1745071548866736171
new_line
text
- Frankenmerge terribly sucked with GSM8K. It seems that adding some Mistral 0.1 layers actually degraded the performance a lot  - this is worse than even 0.1!
- Frankenmerge terribly sucked with GSM8K. It seems that adding some Mistral 0.1 layers actually degraded the performance a lot  - this is worse than even 0.1!
new_line
text
- Otherwise, frankenmerge was decent across HellaSwag, MMLU, and specially TruthfulQA
- Otherwise, frankenmerge was decent across HellaSwag, MMLU, and specially TruthfulQA
new_line
text
- MoE is using random gating, so I expected something right in between 0.1 and 0.2, which was the case
- MoE is using random gating, so I expected something right in between 0.1 and 0.2, which was the case
new_line
new_line
text
What do I do with this?
What do I do with this?
new_line
text
Not sure tbh! I think doing proper MT bench evals would be nice. I also think all of us should give a nice GH star to mergekit because it's awesome. I would love to have the time to do end-to-end ablation studies, but cool new things are coming up. Let me know if you have any thoughts in the results
Not sure tbh! I think doing proper MT bench evals would be nice. I also think all of us should give a nice GH star to mergekit because it's awesome. I would love to have the time to do end-to-end ablation studies, but cool new things are coming up. Let me know if you have any thoughts in the results
I finished my model merging experiment day.🤗I would love your thoughts on this.

What did I do? I merged Mistral Instruct 0.1 and 0.2 models using different merging techniques:
- SLERP: linear interpolation (most popular method)
- MoE: replace some forward layers with MoE layers; using a random gate for now
- Frankenmerge: also known as passthrough, but that isn't very cool. It concatenates some specified layers ending in different numbers of params. In my case, I went from 7B to 9B.

Note: merging is not building an ensemble of models. You can read more about merging techniques at https://huggingface.co/blog/mlabonne/merge-models

Results
I built the 3 models using mergekit (running in an HF Space) - took less than an hour to do the three) https://huggingface.co/collections/osanseviero/mistral-instruct-merges-659ebf35ca0781acdb86bb0a

I'm doing a quick check with the OpenLLM Leaderboard. 
🚨The OpenLLM Leaderboard is more suitable for pre-trained models than instruct models, but I still thought it would be interesting to look at the insights🚨

You can look at the attached image. Some interesting things
- All three models performed somewhere between 0.1 and 0.2 - congrats to the 140 people who got it right in https://twitter.com/osanseviero/status/1745071548866736171
- Frankenmerge terribly sucked with GSM8K. It seems that adding some Mistral 0.1 layers actually degraded the performance a lot  - this is worse than even 0.1!
- Otherwise, frankenmerge was decent across HellaSwag, MMLU, and specially TruthfulQA
- MoE is using random gating, so I expected something right in between 0.1 and 0.2, which was the case

What do I do with this?
Not sure tbh! I think doing proper MT bench evals would be nice. I also think all of us should give a nice GH star to mergekit because it's awesome. I would love to have the time to do end-to-end ablation studies, but cool new things are coming up. Let me know if you have any thoughts in the results
❤️
davanstrien merve mlabonne mrm8488 abhibisht89 gblazex eitanturok detakarang dvilasuero victor julien-c kgourgou thomwolf mwitiderrick not-lain pcuenq marcsun13 clem chansung radames lysandre KnutJaegersberg toshas Mohammadreza m-ric ernestp56 wenqiglantz bhadresh-savani Dlbk nouamanetazi samusenps Csplk Warung jiandong chrissarmstrong sbrandeis Haleshot
🤯
davanstrien pprp dvilasuero julien-c not-lain mrfakename clem chansung lysandre KnutJaegersberg DrishtiSharma nouamanetazi macadeliccc
🤗
mrm8488 julien-c not-lain clem chansung Haleshot
https://cdn-avatars.huggingface.co/v1/production/uploads/6032802e1f993496bc14d9e3/w6hr-DEQot4VVkoyRIBiy.png
Omar Sanseviero
osanseviero
user
https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/JtUGAwVh_4cDEsjNcfpye.png
Maxime Labonne
mlabonne
user
https://cdn-avatars.huggingface.co/v1/production/uploads/1649681653581-5f7fbd813e94f16a85448745.jpeg
Sayak Paul
sayakpaul
user
https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/NQtzmrDdbG0H8qkZvRyGk.jpeg
Julien Chaumond
julien-c
user
https://cdn-avatars.huggingface.co/v1/production/uploads/63f140e52f7c0152e87223f6/1Ml4EXu_ER4R16ACOTVwW.jpeg
Jaime Dols Duxans
duxans
user
/avatars/ca5f5066924ccb4e7f02471c01e49478.svg
Aiswarya Sankar
aiswaryasankar
user