text
A Little guide to building Large Language Models in 2024
A Little guide to building Large Language Models in 2024
new_line
new_line
text
This is a post-recording of a 75min lecture I gave two weeks ago on how to train a LLM from scratch in 2024. I tried to keep it short and comprehensive – focusing on concepts that are crucial for training good LLM but often hidden in tech reports.
This is a post-recording of a 75min lecture I gave two weeks ago on how to train a LLM from scratch in 2024. I tried to keep it short and comprehensive – focusing on concepts that are crucial for training good LLM but often hidden in tech reports.
new_line
new_line
text
In the lecture, I introduce the students to all the important concepts/tools/techniques for training good performance LLM:
In the lecture, I introduce the students to all the important concepts/tools/techniques for training good performance LLM:
new_line
text
* finding, preparing and evaluating web scale data
* finding, preparing and evaluating web scale data
new_line
text
* understanding model parallelism and efficient training
* understanding model parallelism and efficient training
new_line
text
* fine-tuning/aligning models
* fine-tuning/aligning models
new_line
text
* fast inference
* fast inference
new_line
new_line
text
There is of course many things and details missing and that I should have added to it, don't hesitate to tell me you're most frustrating omission and I'll add it in a future part. In particular I think I'll add more focus on how to filter topics well and extensively and maybe more practical anecdotes and details.
There is of course many things and details missing and that I should have added to it, don't hesitate to tell me you're most frustrating omission and I'll add it in a future part. In particular I think I'll add more focus on how to filter topics well and extensively and maybe more practical anecdotes and details.
new_line
new_line
text
Now that I recorded it I've been thinking this could be part 1 of a two-parts series with a 2nd fully hands-on video on how to run all these steps with some libraries and recipes we've released recently at HF around LLM training (and could be easily adapted to your other framework anyway):
Now that I recorded it I've been thinking this could be part 1 of a two-parts series with a 2nd fully hands-on video on how to run all these steps with some libraries and recipes we've released recently at HF around LLM training (and could be easily adapted to your other framework anyway):
new_line
text
*
*
inline_code
`datatrove`
datatrove
text
 for all things web-scale data preparation: 
 for all things web-scale data preparation: 
link
https://github.com/huggingface/datatrove
https://github.com/huggingface/datatrove
new_line
text
*
*
inline_code
`nanotron`
nanotron
text
 for lightweight 4D parallelism LLM training: 
 for lightweight 4D parallelism LLM training: 
link
https://github.com/huggingface/nanotron
https://github.com/huggingface/nanotron
new_line
text
*
*
inline_code
`lighteval`
lighteval
text
 for in-training fast parallel LLM evaluations: 
 for in-training fast parallel LLM evaluations: 
link
https://github.com/huggingface/lighteval
https://github.com/huggingface/lighteval
new_line
new_line
text
Here is the link to watch the lecture on Youtube: 
Here is the link to watch the lecture on Youtube: 
link
https://www.youtube.com/watch?v=2-SPH9hIKT8
https://www.youtube.com/watch?v=2-SPH9hIKT8
new_line
text
And here is the link to the Google slides: 
And here is the link to the Google slides: 
link
https://docs.google.com/presentation/d/1IkzESdOwdmwvPxIELYJi8--K3EZ98_cL6c5ZcLKSyVg/edit#slide=id.p
https://docs.google.com/presentation/d/1IkzESdOwdmwvPxIELYJi8--K3EZ98_cL6c5ZcLKSyVg/edit#slide=id.p
new_line
new_line
text
Enjoy and happy to hear feedback on it and what to add, correct, extend in a second part.
Enjoy and happy to hear feedback on it and what to add, correct, extend in a second part.
A Little guide to building Large Language Models in 2024

This is a post-recording of a 75min lecture I gave two weeks ago on how to train a LLM from scratch in 2024. I tried to keep it short and comprehensive – focusing on concepts that are crucial for training good LLM but often hidden in tech reports.

In the lecture, I introduce the students to all the important concepts/tools/techniques for training good performance LLM:
* finding, preparing and evaluating web scale data
* understanding model parallelism and efficient training
* fine-tuning/aligning models
* fast inference

There is of course many things and details missing and that I should have added to it, don't hesitate to tell me you're most frustrating omission and I'll add it in a future part. In particular I think I'll add more focus on how to filter topics well and extensively and maybe more practical anecdotes and details.

Now that I recorded it I've been thinking this could be part 1 of a two-parts series with a 2nd fully hands-on video on how to run all these steps with some libraries and recipes we've released recently at HF around LLM training (and could be easily adapted to your other framework anyway):
*`datatrove` for all things web-scale data preparation: https://github.com/huggingface/datatrove
*`nanotron` for lightweight 4D parallelism LLM training: https://github.com/huggingface/nanotron
*`lighteval` for in-training fast parallel LLM evaluations: https://github.com/huggingface/lighteval

Here is the link to watch the lecture on Youtube: https://www.youtube.com/watch?v=2-SPH9hIKT8
And here is the link to the Google slides: https://docs.google.com/presentation/d/1IkzESdOwdmwvPxIELYJi8--K3EZ98_cL6c5ZcLKSyVg/edit#slide=id.p

Enjoy and happy to hear feedback on it and what to add, correct, extend in a second part.
❤️
clem osanseviero giux78 samusenps dillfrescott mohammedbriman reach-vb rwightman Venkman42 gusthema 9bow amenur lisuizhe The-Great-Genius danja drFarid MicheleStingo chkla Skier8402 Artples alemiaschi alirezamsh catastropiyush dlicari iproskurina vikas mkhalifa mertege ianlee1996 LiuPengNGP salkhon jdzw2014 serdarcaglar ma7583 derek-thomas fpadovani zackli4ai Davidqian123 JoyboyBrian
🔥
clem osanseviero giux78 nikgr taufiqdp tvergho dillfrescott noh-au-00 reach-vb ajibawa-2023 wip87 thisisanshgupta RichardForests Andron00e 9bow lisuizhe santyzenith yusukewm Ashu212 sid-cypher davzoku gigant lorinma serdarcaglar dnabanita7 derek-thomas zackli4ai Davidqian123 JoyboyBrian
👍
peilin23 dillfrescott reach-vb 9bow jpoag lisuizhe shiv2050 ssml2050 dsindex ad1555 derek-thomas zackli4ai Davidqian123 alexchen4ai JoyboyBrian
🚀
JoyboyBrian
/avatars/cc5c07cc99c647da19cff5219f43869d.svg
noh au
noh-au-00
user
https://cdn-avatars.huggingface.co/v1/production/uploads/651baa9837fecec1fe84034d/bnbi-vyKcbU5OOnqvJpgI.jpeg
Piyush Maharana
catastropiyush
user