text
🚀 Good folks at 
🚀 Good folks at 
mention
@nvidia
nvidia
text
 just dropped: "ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities" 🧠💡
 just dropped: "ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities" 🧠💡
new_line
new_line
text
In the past few months, the open LLM community has made significant progress in releasing open models (Llama-3-70B-Instruct (
In the past few months, the open LLM community has made significant progress in releasing open models (Llama-3-70B-Instruct (
mention
@Meta
Meta
text
 -AI) 🦙, QWen2-72BInstruct (
 -AI) 🦙, QWen2-72BInstruct (
mention
@AlibabaGroup
AlibabaGroup
text
 ) 🌐, Nemotron-4-340B-Instruct (
 ) 🌐, Nemotron-4-340B-Instruct (
mention
@nvidia
nvidia
text
 ) ⚙️, and Mixtral-8x22BInstruct-v0.1 (
 ) ⚙️, and Mixtral-8x22BInstruct-v0.1 (
mention
@MistralAI
MistralAI
text
 ) 🌪️) that are at par with proprietary models! 📈
 ) 🌪️) that are at par with proprietary models! 📈
new_line
new_line
text
But top models like GPT-4 are still outperforming them in certain domains! 🔝💪
But top models like GPT-4 are still outperforming them in certain domains! 🔝💪
new_line
new_line
text
This led us to having domain-focused open-LLMs (DeepSeek-Coder-V2 for coding and math 👨‍💻➕, ChatQA 1.5 for conversational QA and retrieval-augmented generation (RAG) 💬🔍, and InternVL 1.5 for vision-language tasks 🖼️🗣️)
This led us to having domain-focused open-LLMs (DeepSeek-Coder-V2 for coding and math 👨‍💻➕, ChatQA 1.5 for conversational QA and retrieval-augmented generation (RAG) 💬🔍, and InternVL 1.5 for vision-language tasks 🖼️🗣️)
new_line
new_line
text
The challenge that ChatQA 2 focuses on is of context length and RAG! 📏🔗
The challenge that ChatQA 2 focuses on is of context length and RAG! 📏🔗
new_line
new_line
text
These are the two capabilities essential for LLMs to process large volumes of information that cannot fit into a single prompt and are complementary to each other, depending on the downstream tasks and computational budgets. 🧩📊
These are the two capabilities essential for LLMs to process large volumes of information that cannot fit into a single prompt and are complementary to each other, depending on the downstream tasks and computational budgets. 🧩📊
new_line
new_line
text
The solution is a detailed continued training recipe to extend the context window of Llama3-70B-base from 8K to 128K tokens, along with a three-stage instruction tuning process to enhance the model's instruction-following, RAG performance, and long-context understanding capabilities. 🔄🔧
The solution is a detailed continued training recipe to extend the context window of Llama3-70B-base from 8K to 128K tokens, along with a three-stage instruction tuning process to enhance the model's instruction-following, RAG performance, and long-context understanding capabilities. 🔄🔧
new_line
new_line
text
📄 Paper: 
📄 Paper: 
resource
https://huggingface.co/papers/2407.14482
paper
2407.14482
https://huggingface.co/papers/2407.14482
ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG
  Capabilities (2407.14482)
new_line
new_line
text
The interesting thing to notice from benchmarks was, how good QWen 2 is out of the box! 👏✨
The interesting thing to notice from benchmarks was, how good QWen 2 is out of the box! 👏✨
🚀 Good folks at @nvidia just dropped: "ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities" 🧠💡

In the past few months, the open LLM community has made significant progress in releasing open models (Llama-3-70B-Instruct (@Meta -AI) 🦙, QWen2-72BInstruct (@AlibabaGroup ) 🌐, Nemotron-4-340B-Instruct (@nvidia ) ⚙️, and Mixtral-8x22BInstruct-v0.1 (@MistralAI ) 🌪️) that are at par with proprietary models! 📈

But top models like GPT-4 are still outperforming them in certain domains! 🔝💪

This led us to having domain-focused open-LLMs (DeepSeek-Coder-V2 for coding and math 👨‍💻➕, ChatQA 1.5 for conversational QA and retrieval-augmented generation (RAG) 💬🔍, and InternVL 1.5 for vision-language tasks 🖼️🗣️)

The challenge that ChatQA 2 focuses on is of context length and RAG! 📏🔗

These are the two capabilities essential for LLMs to process large volumes of information that cannot fit into a single prompt and are complementary to each other, depending on the downstream tasks and computational budgets. 🧩📊

The solution is a detailed continued training recipe to extend the context window of Llama3-70B-base from 8K to 128K tokens, along with a three-stage instruction tuning process to enhance the model's instruction-following, RAG performance, and long-context understanding capabilities. 🔄🔧

📄 Paper: https://huggingface.co/papers/2407.14482

The interesting thing to notice from benchmarks was, how good QWen 2 is out of the box! 👏✨
👀
louisbrulenaudet