text
Reflection-Llama-3.1-70B burst onto the scene, surprising everyone! It claimed to outperform others with its novel Reflection-Tuning technique, promising not just to match but to surpass the likes of Claude 3.5 and GPT-4o, leveraging its 70 billion parameters to redefine what open-source could achieve.
Reflection-Llama-3.1-70B burst onto the scene, surprising everyone! It claimed to outperform others with its novel Reflection-Tuning technique, promising not just to match but to surpass the likes of Claude 3.5 and GPT-4o, leveraging its 70 billion parameters to redefine what open-source could achieve.
new_line
new_line
text
And now, everything is crumbling!
And now, everything is crumbling!
new_line
new_line
text
The model's performance metrics, especially its 99.2% accuracy on the high school math dataset GSM 8K, have raised eyebrows. While it looked like a valedictorian, based on the open weights, it hardly performs like one.
The model's performance metrics, especially its 99.2% accuracy on the high school math dataset GSM 8K, have raised eyebrows. While it looked like a valedictorian, based on the open weights, it hardly performs like one.
new_line
new_line
text
The model card in the Transformers behaves as Llama 3 and not 3.1.
The model card in the Transformers behaves as Llama 3 and not 3.1.
new_line
new_line
text
While the weights were released publicly, they are having issues aligning with the claims. The tuning has been restarted, and the author claims to upload the updated weights soon!
While the weights were released publicly, they are having issues aligning with the claims. The tuning has been restarted, and the author claims to upload the updated weights soon!
new_line
new_line
text
And the big one: the black-boxed API shared is not at all like the open weights. Even more, when pushed hard, the API endpoint claims to be an LLM by Anthropic!
And the big one: the black-boxed API shared is not at all like the open weights. Even more, when pushed hard, the API endpoint claims to be an LLM by Anthropic!
new_line
new_line
text
But you might ask, didn't this model beat Anthropic Claude 3.5? Yes, it did.
But you might ask, didn't this model beat Anthropic Claude 3.5? Yes, it did.
new_line
new_line
text
So, did Claude 3.5 beat Claude 3.5? No, the benchmark is zero-shot, and the claims are that the results are not under zero-shot but under CoT/few-shot!
So, did Claude 3.5 beat Claude 3.5? No, the benchmark is zero-shot, and the claims are that the results are not under zero-shot but under CoT/few-shot!
new_line
new_line
text
And to top it all off, the reflecting back idea is not new. But I don't think that's a big deal.
And to top it all off, the reflecting back idea is not new. But I don't think that's a big deal.
new_line
new_line
text
I took some time to look through everything, and now, once tested, this model looks to be worse than Llama 3.1 70B
I took some time to look through everything, and now, once tested, this model looks to be worse than Llama 3.1 70B
new_line
new_line
text
I still believe the Reflection-Tuning technique is promising. These are the papers discussing its efficacy:
I still believe the Reflection-Tuning technique is promising. These are the papers discussing its efficacy:
new_line
text
- "Think Before You Speak: Training Language Models With Pause Tokens"
- "Think Before You Speak: Training Language Models With Pause Tokens"
new_line
text
- "Reflection-Tuning: Data Recycling Improves LLM Instruction-Tuning"
- "Reflection-Tuning: Data Recycling Improves LLM Instruction-Tuning"
new_line
new_line
text
PS: Matt Shumer/@mattshumer_ (Twitter Handle) (Reflection-Llama-3.1-70B creator) is a great researcher. Let's wait for his updated weights!
PS: Matt Shumer/@mattshumer_ (Twitter Handle) (Reflection-Llama-3.1-70B creator) is a great researcher. Let's wait for his updated weights!
new_line
new_line
text
Great YT video: 
Great YT video: 
link
https://youtu.be/Xtr_Ll_A9ms
https://youtu.be/Xtr_Ll_A9ms
new_line
new_line
text
Hugging Face Clem Delangue 🤗?
Hugging Face Clem Delangue 🤗?
new_line
text
Can you please help here if possible? This will be the pinnacle of open-source!
Can you please help here if possible? This will be the pinnacle of open-source!
Reflection-Llama-3.1-70B burst onto the scene, surprising everyone! It claimed to outperform others with its novel Reflection-Tuning technique, promising not just to match but to surpass the likes of Claude 3.5 and GPT-4o, leveraging its 70 billion parameters to redefine what open-source could achieve.

And now, everything is crumbling!

The model's performance metrics, especially its 99.2% accuracy on the high school math dataset GSM 8K, have raised eyebrows. While it looked like a valedictorian, based on the open weights, it hardly performs like one.

The model card in the Transformers behaves as Llama 3 and not 3.1.

While the weights were released publicly, they are having issues aligning with the claims. The tuning has been restarted, and the author claims to upload the updated weights soon!

And the big one: the black-boxed API shared is not at all like the open weights. Even more, when pushed hard, the API endpoint claims to be an LLM by Anthropic!

But you might ask, didn't this model beat Anthropic Claude 3.5? Yes, it did.

So, did Claude 3.5 beat Claude 3.5? No, the benchmark is zero-shot, and the claims are that the results are not under zero-shot but under CoT/few-shot!

And to top it all off, the reflecting back idea is not new. But I don't think that's a big deal.

I took some time to look through everything, and now, once tested, this model looks to be worse than Llama 3.1 70B

I still believe the Reflection-Tuning technique is promising. These are the papers discussing its efficacy:
- "Think Before You Speak: Training Language Models With Pause Tokens"
- "Reflection-Tuning: Data Recycling Improves LLM Instruction-Tuning"

PS: Matt Shumer/@mattshumer_ (Twitter Handle) (Reflection-Llama-3.1-70B creator) is a great researcher. Let's wait for his updated weights!

Great YT video: https://youtu.be/Xtr_Ll_A9ms

Hugging Face Clem Delangue 🤗?
Can you please help here if possible? This will be the pinnacle of open-source!
👀
John6666 comarproject danielus louisbrulenaudet
😔
Forbu14
https://cdn-avatars.huggingface.co/v1/production/uploads/64b795b275e4fe20e9fc5aca/v5AWl4XUNF54_0hyRXqBc.jpeg
Ayaan Sharif
Ayaan-Sharif
user