text
After spending some time practicing tokenization, I have come to realize that the difficulties we face in understanding each other are analogous to the challenges faced by LLMs in processing and interpreting tokens - as in untrained tokens lead to out of distribution qualms.
After spending some time practicing tokenization, I have come to realize that the difficulties we face in understanding each other are analogous to the challenges faced by LLMs in processing and interpreting tokens - as in untrained tokens lead to out of distribution qualms.
new_line
new_line
text
One could think of how we understand as a process that involves trained tokens (known/learned facts) grappling with prompts/tweets/lessons from someone else. This process is distinct for each person - with unique encoding, decoding, merging and splitting patterns. 
One could think of how we understand as a process that involves trained tokens (known/learned facts) grappling with prompts/tweets/lessons from someone else. This process is distinct for each person - with unique encoding, decoding, merging and splitting patterns. 
new_line
new_line
text
This distinction might as well be categorized in gpt levels lol, which brings the question what level of tokenizer are you?  GPT-2, GPT-3, GPT-4 or GPT-4o tokenizer:)
This distinction might as well be categorized in gpt levels lol, which brings the question what level of tokenizer are you?  GPT-2, GPT-3, GPT-4 or GPT-4o tokenizer:)
new_line
new_line
text
Papers:
Papers:
new_line
text
- Neural Machine Translation of Rare Words with Subword Units (
- Neural Machine Translation of Rare Words with Subword Units (
link
https://arxiv.org/abs/1508.07909
https://arxiv.org/abs/1508.07909
text
)
)
new_line
text
- Learning to Compress Prompts with Gist Tokens
- Learning to Compress Prompts with Gist Tokens
new_line
text
(
(
link
https://arxiv.org/abs/2304.08467
https://arxiv.org/abs/2304.08467
text
)
)
new_line
text
- Language Models are Few-Shot Learners
- Language Models are Few-Shot Learners
new_line
text
(
(
link
https://arxiv.org/abs/2005.14165
https://arxiv.org/abs/2005.14165
text
)
)
new_line
text
- Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models
- Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models
new_line
text
(
(
link
https://arxiv.org/abs/2405.05417
https://arxiv.org/abs/2405.05417
text
)
)
new_line
text
- Language Models are Unsupervised Multitask Learners 
- Language Models are Unsupervised Multitask Learners 
new_line
new_line
text
Code: 
Code: 
new_line
link
https://github.com/karpathy/minbpe
https://github.com/karpathy/minbpe
new_line
link
https://github.com/openai/tiktoken
https://github.com/openai/tiktoken
new_line
link
https://github.com/openai/gpt-2
https://github.com/openai/gpt-2
new_line
link
https://github.com/google/sentencepiece
https://github.com/google/sentencepiece
new_line
After spending some time practicing tokenization, I have come to realize that the difficulties we face in understanding each other are analogous to the challenges faced by LLMs in processing and interpreting tokens - as in untrained tokens lead to out of distribution qualms.

One could think of how we understand as a process that involves trained tokens (known/learned facts) grappling with prompts/tweets/lessons from someone else. This process is distinct for each person - with unique encoding, decoding, merging and splitting patterns. 

This distinction might as well be categorized in gpt levels lol, which brings the question what level of tokenizer are you?  GPT-2, GPT-3, GPT-4 or GPT-4o tokenizer:)

Papers:
- Neural Machine Translation of Rare Words with Subword Units (https://arxiv.org/abs/1508.07909)
- Learning to Compress Prompts with Gist Tokens
(https://arxiv.org/abs/2304.08467)
- Language Models are Few-Shot Learners
(https://arxiv.org/abs/2005.14165)
- Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models
(https://arxiv.org/abs/2405.05417)
- Language Models are Unsupervised Multitask Learners 

Code: 
https://github.com/karpathy/minbpe
https://github.com/openai/tiktoken
https://github.com/openai/gpt-2
https://github.com/google/sentencepiece

❤️
LeroyDyer fuzzy-g
👀
LeroyDyer
😎
LeroyDyer
https://cdn-avatars.huggingface.co/v1/production/uploads/65d883893a52cd9bcd8ab7cf/tRsCJlHNZo1D02kBTmfy9.jpeg
leroy Samuel Dyer
LeroyDyer
user
https://cdn-avatars.huggingface.co/v1/production/uploads/6438a9027de34e8ea7e4b257/vib8QSd1AWMr_bR9ig_xJ.jpeg
Jaward Sesay
Jaward
user
/avatars/744eddaa7dfc34a57df9ce32a78059a0.svg
Tyrone Pierce
piercyy
user