text
🔥 Introducing "Writing in the Margins (WiM)" - better inference pattern for long context LLMs that solves the Lost-in-the-Middle problem 🔥
🔥 Introducing "Writing in the Margins (WiM)" - better inference pattern for long context LLMs that solves the Lost-in-the-Middle problem 🔥
new_line
new_line
text
Paper page: 
Paper page: 
resource
https://huggingface.co/papers/2408.14906
paper
2408.14906
https://huggingface.co/papers/2408.14906
Writing in the Margins: Better Inference Pattern for Long Context
  Retrieval (2408.14906)
new_line
new_line
text
TL;DR
TL;DR
new_line
text
Make your model write "margin notes" as you chunk prefill the KV cache. Then ask it reread all notes before it speaks up.
Make your model write "margin notes" as you chunk prefill the KV cache. Then ask it reread all notes before it speaks up.
new_line
text
Works with humans, works with AI 🤖
Works with humans, works with AI 🤖
new_line
new_line
text
WiM leverages the chunked prefill of the key-value cache, which concurrently generates query-based extractive summaries at each step of the prefill that are subsequently reintegrated at the end of the computation. We term these intermediate outputs “margins”, drawing inspiration from the practice of making margin notes for improved comprehension of long contexts in human reading. We show that this technique, which adds only minimal additional computation, significantly improves LLMs long context reasoning capabilities. 
WiM leverages the chunked prefill of the key-value cache, which concurrently generates query-based extractive summaries at each step of the prefill that are subsequently reintegrated at the end of the computation. We term these intermediate outputs “margins”, drawing inspiration from the practice of making margin notes for improved comprehension of long contexts in human reading. We show that this technique, which adds only minimal additional computation, significantly improves LLMs long context reasoning capabilities. 
new_line
new_line
text
Think: Every chunk has a chance to be attended to/ be at the end of the context at least once. 🎉
Think: Every chunk has a chance to be attended to/ be at the end of the context at least once. 🎉
new_line
new_line
text
📊 Results:
📊 Results:
new_line
text
- An average accuracy boost of 7.5% in multi-hop reasoning tasks like HotpotQA and MultiHop-RAG.
- An average accuracy boost of 7.5% in multi-hop reasoning tasks like HotpotQA and MultiHop-RAG.
new_line
text
- Even a 30% increase in F1-score for summarisation-like tasks (CWE).
- Even a 30% increase in F1-score for summarisation-like tasks (CWE).
new_line
new_line
text
Plus, WiM fits seamlessly into interactive applications (think: progress bar!). It can provide real-time progress updates during data retrieval and integration, making it user-friendly and transparent - a stark contrast to feeding 1mln tokens to an LLMs and waiting 6 min for the first token. 🤯
Plus, WiM fits seamlessly into interactive applications (think: progress bar!). It can provide real-time progress updates during data retrieval and integration, making it user-friendly and transparent - a stark contrast to feeding 1mln tokens to an LLMs and waiting 6 min for the first token. 🤯
new_line
new_line
text
👩‍💻🧑‍💻 Check it out and contribute to our open-source project here: 
👩‍💻🧑‍💻 Check it out and contribute to our open-source project here: 
link
https://github.com/writer/writing-in-the-margins
https://github.com/writer/writing-in-the-margins
new_line
new_line
text
🧠 More about chunked prefill: 
🧠 More about chunked prefill: 
link
https://docs.vllm.ai/en/latest/models/performance.html#chunked-prefill
https://docs.vllm.ai/en/latest/models/performance.html#chunked-prefill
🔥 Introducing "Writing in the Margins (WiM)" - better inference pattern for long context LLMs that solves the Lost-in-the-Middle problem 🔥

Paper page: https://huggingface.co/papers/2408.14906

TL;DR
Make your model write "margin notes" as you chunk prefill the KV cache. Then ask it reread all notes before it speaks up.
Works with humans, works with AI 🤖

WiM leverages the chunked prefill of the key-value cache, which concurrently generates query-based extractive summaries at each step of the prefill that are subsequently reintegrated at the end of the computation. We term these intermediate outputs “margins”, drawing inspiration from the practice of making margin notes for improved comprehension of long contexts in human reading. We show that this technique, which adds only minimal additional computation, significantly improves LLMs long context reasoning capabilities. 

Think: Every chunk has a chance to be attended to/ be at the end of the context at least once. 🎉

📊 Results:
- An average accuracy boost of 7.5% in multi-hop reasoning tasks like HotpotQA and MultiHop-RAG.
- Even a 30% increase in F1-score for summarisation-like tasks (CWE).

Plus, WiM fits seamlessly into interactive applications (think: progress bar!). It can provide real-time progress updates during data retrieval and integration, making it user-friendly and transparent - a stark contrast to feeding 1mln tokens to an LLMs and waiting 6 min for the first token. 🤯

👩‍💻🧑‍💻 Check it out and contribute to our open-source project here: https://github.com/writer/writing-in-the-margins

🧠 More about chunked prefill: https://docs.vllm.ai/en/latest/models/performance.html#chunked-prefill
🔥
samjulien armiller123 Sri-Vigneshwar-DJ mattsobel victor AdinaY jeffboudier hkproj kiranr andrewrreed John6666 leeloolee clem codelion
❤️
codelion hitchhiker3010
https://cdn-avatars.huggingface.co/v1/production/uploads/666d1e4e2b9e45273912c14a/FffOTN2hceaGWWoGqnJZW.jpeg
Sam Julien
samjulien
user
https://cdn-avatars.huggingface.co/v1/production/uploads/1677134945205-62f32eab52ad88c930bb3f3b.png
Asankhaya Sharma
codelion
user