text
Remember stacking in ensemble ML? 🤔
Remember stacking in ensemble ML? 🤔
new_line
new_line
text
What happens if you do the reverse of that but with LLMs? 🤯
What happens if you do the reverse of that but with LLMs? 🤯
new_line
new_line
text
Basically, MoE created by merging multiple models (instead of being pre-trained like Mixtral)? 🧠
Basically, MoE created by merging multiple models (instead of being pre-trained like Mixtral)? 🧠
new_line
new_line
text
Frankenstein MoE! (not an official name) 🧟‍♂️
Frankenstein MoE! (not an official name) 🧟‍♂️
new_line
new_line
text
That's the new Kraken architecture! 🐙
That's the new Kraken architecture! 🐙
new_line
new_line
text
It uses a sequence classification model to route inputs to the most suitable language model based on the input's characteristics. 🚦
It uses a sequence classification model to route inputs to the most suitable language model based on the input's characteristics. 🚦
new_line
new_line
text
Yup, multiple full-fledged LLMs are loaded into memory, and then a classification layer decides who gets to generate an output! 🎰
Yup, multiple full-fledged LLMs are loaded into memory, and then a classification layer decides who gets to generate an output! 🎰
new_line
new_line
text
Tell me you have too many GPUs without telling me you have too many GPUs! 🖥️🔥
Tell me you have too many GPUs without telling me you have too many GPUs! 🖥️🔥
new_line
new_line
text
Jokes aside, extremely fascinating research but I don't understand why this can't just be a big model with multiple LORA adapters, that can be decided on the fly? 🤷‍♂️
Jokes aside, extremely fascinating research but I don't understand why this can't just be a big model with multiple LORA adapters, that can be decided on the fly? 🤷‍♂️
new_line
new_line
text
Model: 
Model: 
resource
https://huggingface.co/cognitivecomputations/Kraken
model
cognitivecomputations/Kraken
https://huggingface.co/cognitivecomputations/Kraken
new_line
text
Github: 
Github: 
link
https://github.com/cognitivecomputations/kraken
https://github.com/cognitivecomputations/kraken
new_line
Remember stacking in ensemble ML? 🤔

What happens if you do the reverse of that but with LLMs? 🤯

Basically, MoE created by merging multiple models (instead of being pre-trained like Mixtral)? 🧠

Frankenstein MoE! (not an official name) 🧟‍♂️

That's the new Kraken architecture! 🐙

It uses a sequence classification model to route inputs to the most suitable language model based on the input's characteristics. 🚦

Yup, multiple full-fledged LLMs are loaded into memory, and then a classification layer decides who gets to generate an output! 🎰

Tell me you have too many GPUs without telling me you have too many GPUs! 🖥️🔥

Jokes aside, extremely fascinating research but I don't understand why this can't just be a big model with multiple LORA adapters, that can be decided on the fly? 🤷‍♂️

Model: https://huggingface.co/cognitivecomputations/Kraken
Github: https://github.com/cognitivecomputations/kraken

🔥
NickyNicky
https://cdn-avatars.huggingface.co/v1/production/uploads/64da9e4bdafcf7595956abcf/Wf8o2zTUEghptxk-ao_43.jpeg
Alex Sherstinsky
alexsherstinsky
user
/avatars/53ebfcab852efd849a848a26dc65751c.svg
elsatch
elsatch
user