text
📅 Remember when at the beginning of the year 
📅 Remember when at the beginning of the year 
mention
@Google
Google
text
  gave an update on knowledge distillation! Introducing a way of learning from Self-Generated Mistakes?
  gave an update on knowledge distillation! Introducing a way of learning from Self-Generated Mistakes?
new_line
new_line
text
📊 Resulting in significant improvements across tasks:
📊 Resulting in significant improvements across tasks:
new_line
text
- 📄 2.1x in summarization
- 📄 2.1x in summarization
new_line
text
- 🌐 1.7x in translation 
- 🌐 1.7x in translation 
new_line
text
- 🧠 1.9x in reasoning tasks
- 🧠 1.9x in reasoning tasks
new_line
new_line
text
🚀 Well, it looks like Google wasn't messing around! According to the Gemma 2 tech report, knowledge distillation was used to pre-train the 9B model, while the 27B model was pre-trained from scratch.
🚀 Well, it looks like Google wasn't messing around! According to the Gemma 2 tech report, knowledge distillation was used to pre-train the 9B model, while the 27B model was pre-trained from scratch.
new_line
new_line
text
📈 For post-training, the Gemma 2 team generated completions from a stronger teacher model (unspecified in the report, but presumably Gemini Ultra), and then trained the student models on this synthetic data with SFT. This is quite common as seen in many open models, such as Zephyr and OpenHermes.
📈 For post-training, the Gemma 2 team generated completions from a stronger teacher model (unspecified in the report, but presumably Gemini Ultra), and then trained the student models on this synthetic data with SFT. This is quite common as seen in many open models, such as Zephyr and OpenHermes.
new_line
new_line
text
🤔 Sounds too good to be true? These models suffer from distribution mismatch between output sequences seen during training and those generated by the student during inference.
🤔 Sounds too good to be true? These models suffer from distribution mismatch between output sequences seen during training and those generated by the student during inference.
new_line
new_line
text
📰 This is where the January 2024 paper "On-Policy Distillation of Language Models" comes in...
📰 This is where the January 2024 paper "On-Policy Distillation of Language Models" comes in...
new_line
new_line
text
🔍 Gemma 2 team used “on-policy distillation,” where the student generates completions from the SFT prompts. These completions are then used to compute the KL divergence between the teacher’s and student’s logits. By minimizing the KL divergence throughout training, the student learns to model the behavior of the teacher accurately while also minimizing the train-inference mismatch.
🔍 Gemma 2 team used “on-policy distillation,” where the student generates completions from the SFT prompts. These completions are then used to compute the KL divergence between the teacher’s and student’s logits. By minimizing the KL divergence throughout training, the student learns to model the behavior of the teacher accurately while also minimizing the train-inference mismatch.
new_line
new_line
text
📚 Gem🔹 of a blog by 
📚 Gem🔹 of a blog by 
mention
@huggingface
huggingface
text
  uncovering everything Gemma 2: 
  uncovering everything Gemma 2: 
link
https://huggingface.co/blog/gemma2#knowledge-distillation
https://huggingface.co/blog/gemma2#knowledge-distillation
new_line
new_line
text
📄 On-Policy Distillation of Language Models: 
📄 On-Policy Distillation of Language Models: 
resource
https://huggingface.co/papers/2306.13649
paper
2306.13649
https://huggingface.co/papers/2306.13649
On-Policy Distillation of Language Models: Learning from Self-Generated
  Mistakes (2306.13649)
📅 Remember when at the beginning of the year @Google  gave an update on knowledge distillation! Introducing a way of learning from Self-Generated Mistakes?

📊 Resulting in significant improvements across tasks:
- 📄 2.1x in summarization
- 🌐 1.7x in translation 
- 🧠 1.9x in reasoning tasks

🚀 Well, it looks like Google wasn't messing around! According to the Gemma 2 tech report, knowledge distillation was used to pre-train the 9B model, while the 27B model was pre-trained from scratch.

📈 For post-training, the Gemma 2 team generated completions from a stronger teacher model (unspecified in the report, but presumably Gemini Ultra), and then trained the student models on this synthetic data with SFT. This is quite common as seen in many open models, such as Zephyr and OpenHermes.

🤔 Sounds too good to be true? These models suffer from distribution mismatch between output sequences seen during training and those generated by the student during inference.

📰 This is where the January 2024 paper "On-Policy Distillation of Language Models" comes in...

🔍 Gemma 2 team used “on-policy distillation,” where the student generates completions from the SFT prompts. These completions are then used to compute the KL divergence between the teacher’s and student’s logits. By minimizing the KL divergence throughout training, the student learns to model the behavior of the teacher accurately while also minimizing the train-inference mismatch.

📚 Gem🔹 of a blog by @huggingface  uncovering everything Gemma 2: https://huggingface.co/blog/gemma2#knowledge-distillation

📄 On-Policy Distillation of Language Models: https://huggingface.co/papers/2306.13649