text
You are all happy 😊 that 
You are all happy 😊 that 
mention
@meta-llama
meta-llama
text
 released Llama 3.
 released Llama 3.
new_line
new_line
text
Then you are sad 😔 that it only has a context length of 8k.
Then you are sad 😔 that it only has a context length of 8k.
new_line
new_line
text
Then you are happy 😄 that you can just scale llama-3 PoSE to 96k without training, only needing to modify max_position_embeddings and rope_theta.
Then you are happy 😄 that you can just scale llama-3 PoSE to 96k without training, only needing to modify max_position_embeddings and rope_theta.
new_line
new_line
text
But then you are sad 😢 it only improves the model's long-context retrieval performance (i.e., finding needles) while hardly improving its long-context utilization capability (doing QA and summarization).
But then you are sad 😢 it only improves the model's long-context retrieval performance (i.e., finding needles) while hardly improving its long-context utilization capability (doing QA and summarization).
new_line
new_line
text
But then you are happy 😁 that the 
But then you are happy 😁 that the 
new_line
mention
@GradientsTechnologies
GradientsTechnologies
text
 community has released the long-context Llama-3-8B-Instruct-262K with long context (262k-1M+).
 community has released the long-context Llama-3-8B-Instruct-262K with long context (262k-1M+).
new_line
new_line
text
Now we have another paper "Extending Llama-3's Context Ten-Fold Overnight" 📜.
Now we have another paper "Extending Llama-3's Context Ten-Fold Overnight" 📜.
new_line
new_line
text
The context length of Llama-3-8B-Instruct is extended from 8K to 80K using QLoRA fine-tuning⚙️.
The context length of Llama-3-8B-Instruct is extended from 8K to 80K using QLoRA fine-tuning⚙️.
new_line
new_line
text
The training cycle is highly efficient, taking "only" 😂 8 hours on a single 8xA800 (80G) GPU machine.
The training cycle is highly efficient, taking "only" 😂 8 hours on a single 8xA800 (80G) GPU machine.
new_line
new_line
text
The model also preserves its original capability over short contexts. ✁
The model also preserves its original capability over short contexts. ✁
new_line
new_line
text
The dramatic context extension is mainly attributed to merely 3.5K synthetic training samples generated by GPT-4.📊
The dramatic context extension is mainly attributed to merely 3.5K synthetic training samples generated by GPT-4.📊
new_line
new_line
text
The paper suggests that the context length could be extended far beyond 80K with more computation resources (😅 GPU-poor).
The paper suggests that the context length could be extended far beyond 80K with more computation resources (😅 GPU-poor).
new_line
new_line
text
The team plans to publicly release all resources, including data, model, data generation pipeline, and training code, to facilitate future research from the ❤️ community.
The team plans to publicly release all resources, including data, model, data generation pipeline, and training code, to facilitate future research from the ❤️ community.
new_line
new_line
text
Paper: 
Paper: 
link
https://arxiv.org/abs/2404.19553
https://arxiv.org/abs/2404.19553
new_line
new_line
text
This is where we are... until next time... 🌟
This is where we are... until next time... 🌟
new_line
new_line
resource
https://huggingface.co/papers/2404.19553
paper
2404.19553
https://huggingface.co/papers/2404.19553
Extending Llama-3's Context Ten-Fold Overnight (2404.19553)
You are all happy 😊 that @meta-llama released Llama 3.

Then you are sad 😔 that it only has a context length of 8k.

Then you are happy 😄 that you can just scale llama-3 PoSE to 96k without training, only needing to modify max_position_embeddings and rope_theta.

But then you are sad 😢 it only improves the model's long-context retrieval performance (i.e., finding needles) while hardly improving its long-context utilization capability (doing QA and summarization).

But then you are happy 😁 that the 
@GradientsTechnologies community has released the long-context Llama-3-8B-Instruct-262K with long context (262k-1M+).

Now we have another paper "Extending Llama-3's Context Ten-Fold Overnight" 📜.

The context length of Llama-3-8B-Instruct is extended from 8K to 80K using QLoRA fine-tuning⚙️.

The training cycle is highly efficient, taking "only" 😂 8 hours on a single 8xA800 (80G) GPU machine.

The model also preserves its original capability over short contexts. ✁

The dramatic context extension is mainly attributed to merely 3.5K synthetic training samples generated by GPT-4.📊

The paper suggests that the context length could be extended far beyond 80K with more computation resources (😅 GPU-poor).

The team plans to publicly release all resources, including data, model, data generation pipeline, and training code, to facilitate future research from the ❤️ community.

Paper: https://arxiv.org/abs/2404.19553

This is where we are... until next time... 🌟

https://huggingface.co/papers/2404.19553
❤️
tasmani osanseviero Kukedlc Beeezy PrasannaAtlas not-lain prashant182 jrell JoPmt
🤗
asif10018 osanseviero Kukedlc Beeezy not-lain prashant182 Ramikan-BR prsyahmi
🚀
osanseviero Kukedlc Beeezy not-lain prashant182 Ramikan-BR
👀
Ramikan-BR
👍
Ramikan-BR