text
Ok, you're finally convinced that synthetic data works... ⚗️
Ok, you're finally convinced that synthetic data works... ⚗️
new_line
new_line
text
𝐍𝐨𝐰 𝐲𝐨𝐮 𝐰𝐚𝐧𝐭 𝐭𝐨 𝐠𝐞𝐧𝐞𝐫𝐚𝐭𝐞 𝐚𝐧 𝐢𝐧𝐬𝐭𝐫𝐮𝐜𝐭𝐢𝐨𝐧 𝐝𝐚𝐭𝐚𝐬𝐞𝐭 𝐟𝐨𝐫 𝐟𝐢𝐧𝐞-𝐭𝐮𝐧𝐢𝐧𝐠 𝐢𝐧 𝐚 𝐥𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐨𝐭𝐡𝐞𝐫 𝐭𝐡𝐚𝐧 𝐄𝐧𝐠𝐥𝐢𝐬𝐡.
𝐍𝐨𝐰 𝐲𝐨𝐮 𝐰𝐚𝐧𝐭 𝐭𝐨 𝐠𝐞𝐧𝐞𝐫𝐚𝐭𝐞 𝐚𝐧 𝐢𝐧𝐬𝐭𝐫𝐮𝐜𝐭𝐢𝐨𝐧 𝐝𝐚𝐭𝐚𝐬𝐞𝐭 𝐟𝐨𝐫 𝐟𝐢𝐧𝐞-𝐭𝐮𝐧𝐢𝐧𝐠 𝐢𝐧 𝐚 𝐥𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐨𝐭𝐡𝐞𝐫 𝐭𝐡𝐚𝐧 𝐄𝐧𝐠𝐥𝐢𝐬𝐡.
new_line
text
But how do you get started?
But how do you get started?
new_line
new_line
text
I explore how to do this with Magpie in my new article
I explore how to do this with Magpie in my new article
new_line
link
https://huggingface.co/blog/anakin87/multilingual-magpie
https://huggingface.co/blog/anakin87/multilingual-magpie
new_line
new_line
text
---
---
new_line
new_line
text
🐦‍⬛ 𝐖𝐡𝐚𝐭 𝐢𝐬 𝐌𝐚𝐠𝐩𝐢𝐞?
🐦‍⬛ 𝐖𝐡𝐚𝐭 𝐢𝐬 𝐌𝐚𝐠𝐩𝐢𝐞?
new_line
new_line
text
It's a recent technique for creating synthetic instruction datasets.
It's a recent technique for creating synthetic instruction datasets.
new_line
new_line
text
Magpie is based on a simple but ingenious idea 👇
Magpie is based on a simple but ingenious idea 👇
new_line
text
if you prompt an instruction-tuned model with a pre-query template, you can make it generate a plausible user query/instruction
if you prompt an instruction-tuned model with a pre-query template, you can make it generate a plausible user query/instruction
new_line
new_line
text
Here's an example:
Here's an example:
new_line
text
model: Llama-3-8B-Instruct
model: Llama-3-8B-Instruct
new_line
text
pre-query template: "<|begin_of_text|><|start_header_id|>user<|end_header_id|>"
pre-query template: "<|begin_of_text|><|start_header_id|>user<|end_header_id|>"
new_line
text
generated user instruction: "What are some of the responsibilities of a commercial pilot?"
generated user instruction: "What are some of the responsibilities of a commercial pilot?"
new_line
new_line
text
You can then feed this instruction back into the same model to get the assistant response.
You can then feed this instruction back into the same model to get the assistant response.
new_line
new_line
text
By repeating this process, it's possible to generate large synthetic datasets with relatively little effort.
By repeating this process, it's possible to generate large synthetic datasets with relatively little effort.
new_line
new_line
text
🪄 The authors demonstrate that using these datasets for Supervised Fine Tuning (SFT) can yield strong performance, even competitive with the original instruct model.
🪄 The authors demonstrate that using these datasets for Supervised Fine Tuning (SFT) can yield strong performance, even competitive with the original instruct model.
new_line
new_line
new_line
text
🧗𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐧𝐠 𝐧𝐨𝐧-𝐄𝐧𝐠𝐥𝐢𝐬𝐡 𝐝𝐚𝐭𝐚
🧗𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐧𝐠 𝐧𝐨𝐧-𝐄𝐧𝐠𝐥𝐢𝐬𝐡 𝐝𝐚𝐭𝐚
new_line
new_line
text
Most Language Models are primarily trained on English texts, so they tend to produce data in English.
Most Language Models are primarily trained on English texts, so they tend to produce data in English.
new_line
new_line
text
How can we overcome this?
How can we overcome this?
new_line
new_line
text
Earlier approaches were complex or costly.
Earlier approaches were complex or costly.
new_line
new_line
text
Then 
Then 
mention
@mrm8488
mrm8488
text
  found a simple solution: add the target language to the pre-query template.
  found a simple solution: add the target language to the pre-query template.
new_line
text
For Spanish, the template becomes "<|begin_of_text|><|start_header_id|>user<|end_header_id|>spanish:".
For Spanish, the template becomes "<|begin_of_text|><|start_header_id|>user<|end_header_id|>spanish:".
new_line
new_line
text
This method works for Spanish and German!
This method works for Spanish and German!
new_line
new_line
text
❌ Unfortunately, it does not work well for other languages (🇮🇹, 🇳🇱, ...)
❌ Unfortunately, it does not work well for other languages (🇮🇹, 🇳🇱, ...)
new_line
new_line
text
👇
👇
Ok, you're finally convinced that synthetic data works... ⚗️

𝐍𝐨𝐰 𝐲𝐨𝐮 𝐰𝐚𝐧𝐭 𝐭𝐨 𝐠𝐞𝐧𝐞𝐫𝐚𝐭𝐞 𝐚𝐧 𝐢𝐧𝐬𝐭𝐫𝐮𝐜𝐭𝐢𝐨𝐧 𝐝𝐚𝐭𝐚𝐬𝐞𝐭 𝐟𝐨𝐫 𝐟𝐢𝐧𝐞-𝐭𝐮𝐧𝐢𝐧𝐠 𝐢𝐧 𝐚 𝐥𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐨𝐭𝐡𝐞𝐫 𝐭𝐡𝐚𝐧 𝐄𝐧𝐠𝐥𝐢𝐬𝐡.
But how do you get started?

I explore how to do this with Magpie in my new article
https://huggingface.co/blog/anakin87/multilingual-magpie

---

🐦‍⬛ 𝐖𝐡𝐚𝐭 𝐢𝐬 𝐌𝐚𝐠𝐩𝐢𝐞?

It's a recent technique for creating synthetic instruction datasets.

Magpie is based on a simple but ingenious idea 👇
if you prompt an instruction-tuned model with a pre-query template, you can make it generate a plausible user query/instruction

Here's an example:
model: Llama-3-8B-Instruct
pre-query template: "<|begin_of_text|><|start_header_id|>user<|end_header_id|>"
generated user instruction: "What are some of the responsibilities of a commercial pilot?"

You can then feed this instruction back into the same model to get the assistant response.

By repeating this process, it's possible to generate large synthetic datasets with relatively little effort.

🪄 The authors demonstrate that using these datasets for Supervised Fine Tuning (SFT) can yield strong performance, even competitive with the original instruct model.


🧗𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐧𝐠 𝐧𝐨𝐧-𝐄𝐧𝐠𝐥𝐢𝐬𝐡 𝐝𝐚𝐭𝐚

Most Language Models are primarily trained on English texts, so they tend to produce data in English.

How can we overcome this?

Earlier approaches were complex or costly.

Then @mrm8488  found a simple solution: add the target language to the pre-query template.
For Spanish, the template becomes "<|begin_of_text|><|start_header_id|>user<|end_header_id|>spanish:".

This method works for Spanish and German!

❌ Unfortunately, it does not work well for other languages (🇮🇹, 🇳🇱, ...)

👇
👀
John6666 tiendung djuna jgitsolutions
🤯
pacificg
https://cdn-avatars.huggingface.co/v1/production/uploads/626505d493e0b04d75710566/9rfJc9ORXU9J5a42Ev3v6.png
Stefano Fiorucci
anakin87
user