text
hi everyone!
hi everyone!
new_line
new_line
text
i wanted to share an experiment i did with upcycling phi-3 mini into an moe recently.
i wanted to share an experiment i did with upcycling phi-3 mini into an moe recently.
new_line
text
while benchmarks are definitely within a margin of error and they performed similarly, i think it's an interesting base to try and see if you can improve phi's performance! (maybe looking into 
while benchmarks are definitely within a margin of error and they performed similarly, i think it's an interesting base to try and see if you can improve phi's performance! (maybe looking into 
resource
https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu
dataset
HuggingFaceFW/fineweb-edu
https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu
text
 could be interesting, i also left some other notes if anyone with more compute access wants to try it themselves)
 could be interesting, i also left some other notes if anyone with more compute access wants to try it themselves)
new_line
new_line
text
check it out! 
check it out! 
resource
https://huggingface.co/Fizzarolli/phi3-4x4b-v1
model
Fizzarolli/phi3-4x4b-v1
https://huggingface.co/Fizzarolli/phi3-4x4b-v1
hi everyone!

i wanted to share an experiment i did with upcycling phi-3 mini into an moe recently.
while benchmarks are definitely within a margin of error and they performed similarly, i think it's an interesting base to try and see if you can improve phi's performance! (maybe looking into https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu could be interesting, i also left some other notes if anyone with more compute access wants to try it themselves)

check it out! https://huggingface.co/Fizzarolli/phi3-4x4b-v1
👍
Hastagaras