text
Here is a thought, instead of telling LLMs what to do, show them! 🎭
Here is a thought, instead of telling LLMs what to do, show them! 🎭
new_line
new_line
text
Language models are aligned to emulate the collective voice of many, resulting in outputs that align with no one in particular. 🗣️🌍
Language models are aligned to emulate the collective voice of many, resulting in outputs that align with no one in particular. 🗣️🌍
new_line
new_line
text
DITTO from Stanford University proposes that LLMs can be tuned with less than 10 samples! 🤯
DITTO from Stanford University proposes that LLMs can be tuned with less than 10 samples! 🤯
new_line
new_line
text
What's DITTO? Demonstration ITerated Task Optimization (definitely came up with the acronym first! 😂)
What's DITTO? Demonstration ITerated Task Optimization (definitely came up with the acronym first! 😂)
new_line
new_line
text
Here is the step-by-step implementation: 🛠️
Here is the step-by-step implementation: 🛠️
new_line
new_line
text
Initialization: Start with a reference language model (LM), a set of expert demonstrations, a sample size, and a frequency of sampling. 🏁
Initialization: Start with a reference language model (LM), a set of expert demonstrations, a sample size, and a frequency of sampling. 🏁
new_line
new_line
text
Supervised Fine-Tuning (SFT): Begin by fine-tuning the reference LM on the set of expert demonstrations to create an initial policy P0. 🎚️
Supervised Fine-Tuning (SFT): Begin by fine-tuning the reference LM on the set of expert demonstrations to create an initial policy P0. 🎚️
new_line
new_line
text
Iterative Comparison Sampling: For each iteration t:
Iterative Comparison Sampling: For each iteration t:
new_line
text
Sample multiple completions from the policy Pt for each demonstration to create a new dataset Dt.
Sample multiple completions from the policy Pt for each demonstration to create a new dataset Dt.
new_line
text
Construct a batch of comparisons where the demonstrations are ranked higher than all sampled model outputs from the current and previous iterations. 🔄
Construct a batch of comparisons where the demonstrations are ranked higher than all sampled model outputs from the current and previous iterations. 🔄
new_line
new_line
text
Policy Update:
Policy Update:
new_line
text
Update the policy Pt using a Direct Preference Optimization (DPO) algorithm, which incorporates feedback from the batch of comparisons.
Update the policy Pt using a Direct Preference Optimization (DPO) algorithm, which incorporates feedback from the batch of comparisons.
new_line
text
Increment the iteration and repeat the sampling and updating process until convergence. ⏭️
Increment the iteration and repeat the sampling and updating process until convergence. ⏭️
new_line
new_line
text
Result: The final policy P after sufficient iterations aligns more closely with the expert demonstrations, effectively tuning the LM to reflect user-specific preferences and behaviors. 🎯
Result: The final policy P after sufficient iterations aligns more closely with the expert demonstrations, effectively tuning the LM to reflect user-specific preferences and behaviors. 🎯
new_line
new_line
text
DITTO outperforms few-shot prompting. 🚀
DITTO outperforms few-shot prompting. 🚀
new_line
new_line
text
Paper: 
Paper: 
resource
https://huggingface.co/papers/2406.00888
paper
2406.00888
https://huggingface.co/papers/2406.00888
Show, Don't Tell: Aligning Language Models with Demonstrated Feedback (2406.00888)
text
 📄
 📄
Here is a thought, instead of telling LLMs what to do, show them! 🎭

Language models are aligned to emulate the collective voice of many, resulting in outputs that align with no one in particular. 🗣️🌍

DITTO from Stanford University proposes that LLMs can be tuned with less than 10 samples! 🤯

What's DITTO? Demonstration ITerated Task Optimization (definitely came up with the acronym first! 😂)

Here is the step-by-step implementation: 🛠️

Initialization: Start with a reference language model (LM), a set of expert demonstrations, a sample size, and a frequency of sampling. 🏁

Supervised Fine-Tuning (SFT): Begin by fine-tuning the reference LM on the set of expert demonstrations to create an initial policy P0. 🎚️

Iterative Comparison Sampling: For each iteration t:
Sample multiple completions from the policy Pt for each demonstration to create a new dataset Dt.
Construct a batch of comparisons where the demonstrations are ranked higher than all sampled model outputs from the current and previous iterations. 🔄

Policy Update:
Update the policy Pt using a Direct Preference Optimization (DPO) algorithm, which incorporates feedback from the batch of comparisons.
Increment the iteration and repeat the sampling and updating process until convergence. ⏭️

Result: The final policy P after sufficient iterations aligns more closely with the expert demonstrations, effectively tuning the LM to reflect user-specific preferences and behaviors. 🎯

DITTO outperforms few-shot prompting. 🚀

Paper: https://huggingface.co/papers/2406.00888 📄
🔥
osanseviero GPT007 graphicaldot DavidGF
👍
pduf kap9911
🤝
pduf
😔
pduf
🤗
Tonic