text
"How expensive is it actually to teach a #LanguageModel German through #finetuning 💰💰💰? We get asked this quite often.
"How expensive is it actually to teach a #LanguageModel German through #finetuning 💰💰💰? We get asked this quite often.
new_line
new_line
text
There is no one-size-fits-all answer to this question, as among other factors:
There is no one-size-fits-all answer to this question, as among other factors:
new_line
text
⏹ each fine-tuning is different,
⏹ each fine-tuning is different,
new_line
text
⏹ the hardware used can be a major cost driver,
⏹ the hardware used can be a major cost driver,
new_line
text
⏹ the amount and type of training data can extend the process,
⏹ the amount and type of training data can extend the process,
new_line
text
⏹ and the skills to be trained can increase the difficulty of fine-tuning.
⏹ and the skills to be trained can increase the difficulty of fine-tuning.
new_line
new_line
text
However, we have broken down the costs incurred for our latest fine-tune ( 
However, we have broken down the costs incurred for our latest fine-tune ( 
resource
https://huggingface.co/VAGOsolutions/SauerkrautLM-Qwen-32b
model
VAGOsolutions/SauerkrautLM-Qwen-32b
https://huggingface.co/VAGOsolutions/SauerkrautLM-Qwen-32b
text
)
)
new_line
new_line
new_line
text
Base model: 
Base model: 
resource
https://huggingface.co/Qwen/Qwen1.5-32B
model
Qwen/Qwen1.5-32B
https://huggingface.co/Qwen/Qwen1.5-32B
new_line
text
Fine-Tuning Goal: Train German language
Fine-Tuning Goal: Train German language
new_line
text
Training dataset size: 160,000 SFT data / 110,000 DPO data
Training dataset size: 160,000 SFT data / 110,000 DPO data
new_line
text
Training duration: 72.5 hours (2 epochs SFT / 1 epoch DPO)
Training duration: 72.5 hours (2 epochs SFT / 1 epoch DPO)
new_line
text
GPU: 2x A100 SXM
GPU: 2x A100 SXM
new_line
text
New model: 
New model: 
resource
https://huggingface.co/VAGOsolutions/SauerkrautLM-Qwen-32b
model
VAGOsolutions/SauerkrautLM-Qwen-32b
https://huggingface.co/VAGOsolutions/SauerkrautLM-Qwen-32b
new_line
new_line
text
Total cost: 312 euros 💶
Total cost: 312 euros 💶
new_line
new_line
text
These are quite reasonable training costs considering the model now speaks passable German (previously very broken). Depending on the use case and process requirements, this can even be a real alternative to the costly continuous pre-training of foreign language models.
These are quite reasonable training costs considering the model now speaks passable German (previously very broken). Depending on the use case and process requirements, this can even be a real alternative to the costly continuous pre-training of foreign language models.
"How expensive is it actually to teach a #LanguageModel German through #finetuning 💰💰💰? We get asked this quite often.

There is no one-size-fits-all answer to this question, as among other factors:
⏹ each fine-tuning is different,
⏹ the hardware used can be a major cost driver,
⏹ the amount and type of training data can extend the process,
⏹ and the skills to be trained can increase the difficulty of fine-tuning.

However, we have broken down the costs incurred for our latest fine-tune ( https://huggingface.co/VAGOsolutions/SauerkrautLM-Qwen-32b)


Base model: https://huggingface.co/Qwen/Qwen1.5-32B
Fine-Tuning Goal: Train German language
Training dataset size: 160,000 SFT data / 110,000 DPO data
Training duration: 72.5 hours (2 epochs SFT / 1 epoch DPO)
GPU: 2x A100 SXM
New model: https://huggingface.co/VAGOsolutions/SauerkrautLM-Qwen-32b

Total cost: 312 euros 💶

These are quite reasonable training costs considering the model now speaks passable German (previously very broken). Depending on the use case and process requirements, this can even be a real alternative to the costly continuous pre-training of foreign language models.
🔥
jed-tiotuico osanseviero clem victor fpreiss philipp-zettl joennlae Kearm