text
Researchers from 
Researchers from 
mention
@GoogleDeepMind
GoogleDeepMind
text
 have introduced "Michelangelo" — a novel framework for evaluating large language models on long-context reasoning tasks beyond simple retrieval.
 have introduced "Michelangelo" — a novel framework for evaluating large language models on long-context reasoning tasks beyond simple retrieval.
new_line
new_line
text
They have proposed three minimal tasks to test different aspects of long-context reasoning:
They have proposed three minimal tasks to test different aspects of long-context reasoning:
new_line
text
- Latent List: Tracking a Python list's state over many operations.
- Latent List: Tracking a Python list's state over many operations.
new_line
text
- MRCR: Multi-round coreference resolution in conversations.
- MRCR: Multi-round coreference resolution in conversations.
new_line
text
- IDK: Determining if an answer exists in a long context.
- IDK: Determining if an answer exists in a long context.
new_line
new_line
text
They found significant performance drop-offs before 32K tokens on these tasks, indicating room for improvement in long-context reasoning.
They found significant performance drop-offs before 32K tokens on these tasks, indicating room for improvement in long-context reasoning.
new_line
new_line
text
Here are the key steps for creating the Michelangelo long-context evaluations:
Here are the key steps for creating the Michelangelo long-context evaluations:
new_line
new_line
text
1. Develop the Latent Structure Queries (LSQ) framework:
1. Develop the Latent Structure Queries (LSQ) framework:
new_line
text
- Create a framework for generating long-context evaluations that can be extended arbitrarily in length and complexity.
- Create a framework for generating long-context evaluations that can be extended arbitrarily in length and complexity.
new_line
text
- Ensure the framework measures capabilities beyond simple retrieval.
- Ensure the framework measures capabilities beyond simple retrieval.
new_line
new_line
text
2. Design minimal tasks using the LSQ framework:
2. Design minimal tasks using the LSQ framework:
new_line
text
- Create tasks that test different aspects of long-context reasoning.
- Create tasks that test different aspects of long-context reasoning.
new_line
text
- Ensure tasks are minimally complex while still challenging for current models.
- Ensure tasks are minimally complex while still challenging for current models.
new_line
new_line
text
3. Implement the Latent List task:
3. Implement the Latent List task:
new_line
text
- Create a Python list-based task with operations that modify the list.
- Create a Python list-based task with operations that modify the list.
new_line
text
- Include relevant and irrelevant operations to test model understanding.
- Include relevant and irrelevant operations to test model understanding.
new_line
text
- Develop view operations to query the final state of the list.
- Develop view operations to query the final state of the list.
new_line
new_line
text
4. Implement the Multi-Round Coreference Resolution (MRCR) task:
4. Implement the Multi-Round Coreference Resolution (MRCR) task:
new_line
text
- Generate conversations with user requests and model responses on various topics.
- Generate conversations with user requests and model responses on various topics.
new_line
text
- Place specific requests randomly in the context.
- Place specific requests randomly in the context.
new_line
text
- Require models to reproduce outputs based on queries about the conversation.
- Require models to reproduce outputs based on queries about the conversation.
new_line
new_line
text
5. Implement the IDK task:
5. Implement the IDK task:
new_line
text
- Create contexts with invented stories or information.
- Create contexts with invented stories or information.
new_line
text
- Develop questions that may or may not have answers in the context.
- Develop questions that may or may not have answers in the context.
new_line
text
- Include multiple-choice options, always including "I don't know" as an option.
- Include multiple-choice options, always including "I don't know" as an option.
new_line
new_line
text
More in comments...
More in comments...
Researchers from @GoogleDeepMind have introduced "Michelangelo" — a novel framework for evaluating large language models on long-context reasoning tasks beyond simple retrieval.

They have proposed three minimal tasks to test different aspects of long-context reasoning:
- Latent List: Tracking a Python list's state over many operations.
- MRCR: Multi-round coreference resolution in conversations.
- IDK: Determining if an answer exists in a long context.

They found significant performance drop-offs before 32K tokens on these tasks, indicating room for improvement in long-context reasoning.

Here are the key steps for creating the Michelangelo long-context evaluations:

1. Develop the Latent Structure Queries (LSQ) framework:
- Create a framework for generating long-context evaluations that can be extended arbitrarily in length and complexity.
- Ensure the framework measures capabilities beyond simple retrieval.

2. Design minimal tasks using the LSQ framework:
- Create tasks that test different aspects of long-context reasoning.
- Ensure tasks are minimally complex while still challenging for current models.

3. Implement the Latent List task:
- Create a Python list-based task with operations that modify the list.
- Include relevant and irrelevant operations to test model understanding.
- Develop view operations to query the final state of the list.

4. Implement the Multi-Round Coreference Resolution (MRCR) task:
- Generate conversations with user requests and model responses on various topics.
- Place specific requests randomly in the context.
- Require models to reproduce outputs based on queries about the conversation.

5. Implement the IDK task:
- Create contexts with invented stories or information.
- Develop questions that may or may not have answers in the context.
- Include multiple-choice options, always including "I don't know" as an option.

More in comments...
🧠
lunarflu John6666 bfuzzy1 louisbrulenaudet
🔥
ajibawa-2023 bfuzzy1
😎
pduf
https://cdn-avatars.huggingface.co/v1/production/uploads/662bf5bfe93bb73804ef9344/WXYLnjjJ4SROkoveIi7If.png
Kuldeep Singh Sidhu
singhsidhukuldeep
user