text
Build your own GPT-4 Tokenizer! - 
Build your own GPT-4 Tokenizer! - 
mention
@karpathy
karpathy
text
's minbpe exercise.
's minbpe exercise.
new_line
text
Step 1: BasicTokenizer
Step 1: BasicTokenizer
new_line
text
Got "close" to beating minbpe's train speed :(
Got "close" to beating minbpe's train speed :(
new_line
text
step 2 RegexTokenizer coming soon.
step 2 RegexTokenizer coming soon.
new_line
new_line
text
Notes on lessons learned:
Notes on lessons learned:
new_line
text
- tokenization is the assembly language of LLMs:)
- tokenization is the assembly language of LLMs:)
new_line
text
It's not a healthy choice to code it lol.
It's not a healthy choice to code it lol.
new_line
text
- encoding can literally drive you mad.
- encoding can literally drive you mad.
new_line
text
- merging is where sh*t gets real - moment of truth:)
- merging is where sh*t gets real - moment of truth:)
new_line
text
- training requires precision.
- training requires precision.
new_line
text
- decoding is trivial.
- decoding is trivial.
Build your own GPT-4 Tokenizer! - @karpathy's minbpe exercise.
Step 1: BasicTokenizer
Got "close" to beating minbpe's train speed :(
step 2 RegexTokenizer coming soon.

Notes on lessons learned:
- tokenization is the assembly language of LLMs:)
It's not a healthy choice to code it lol.
- encoding can literally drive you mad.
- merging is where sh*t gets real - moment of truth:)
- training requires precision.
- decoding is trivial.
👀
clem wannaphong victor lunarflu
https://cdn-avatars.huggingface.co/v1/production/uploads/6438a9027de34e8ea7e4b257/vib8QSd1AWMr_bR9ig_xJ.jpeg
Jaward Sesay
Jaward
user