text
# Thoughts on Neural Scaling Laws
# Thoughts on Neural Scaling Laws
new_line
text
When you take a zoomed-out perspective view on the success goals of neural networks, you see they all revolve around the Scaling Laws - empirical observations that performance improves with increased model size, dataset, and compute resources.
When you take a zoomed-out perspective view on the success goals of neural networks, you see they all revolve around the Scaling Laws - empirical observations that performance improves with increased model size, dataset, and compute resources.
new_line
new_line
text
The specifics of how these laws apply, vary for different modalities and architectures. This is notable in the empirical equations used to measure these laws. 
The specifics of how these laws apply, vary for different modalities and architectures. This is notable in the empirical equations used to measure these laws. 
new_line
new_line
text
Yet they all heavily rely on three main factors - Data, Size and Computation. These factors themselves also have sub-dependencies - data size & quality, model size & architecture, num of GPUs & code for compute kernels respectively.
Yet they all heavily rely on three main factors - Data, Size and Computation. These factors themselves also have sub-dependencies - data size & quality, model size & architecture, num of GPUs & code for compute kernels respectively.
new_line
new_line
text
As research in these laws progresses, we begin to see new scaling laws emerge that may apply in much different ways than usual. This is typical in recent local LLMs (Phi-3, Gemma 2B, LLMs in a flash) which shows small sized models with small rich quality data beating large models
As research in these laws progresses, we begin to see new scaling laws emerge that may apply in much different ways than usual. This is typical in recent local LLMs (Phi-3, Gemma 2B, LLMs in a flash) which shows small sized models with small rich quality data beating large models
new_line
new_line
text
I look forward to the singularity moment - when these laws take a full round spin and meet at where it all began:)
I look forward to the singularity moment - when these laws take a full round spin and meet at where it all began:)
new_line
new_line
text
References:
References:
new_line
text
- Scaling Laws for Neural Language Models: 
- Scaling Laws for Neural Language Models: 
link
https://arxiv.org/pdf/2001.08361
https://arxiv.org/pdf/2001.08361
new_line
text
- Scaling Laws for Autoregressive Generative Modeling: 
- Scaling Laws for Autoregressive Generative Modeling: 
link
https://arxiv.org/abs/2010.14701
https://arxiv.org/abs/2010.14701
new_line
text
- LLMs in a flash: 
- LLMs in a flash: 
link
https://arxiv.org/abs/2312.11514
https://arxiv.org/abs/2312.11514
new_line
text
- Phi-3 Technical Report: 
- Phi-3 Technical Report: 
link
https://arxiv.org/abs/2404.14219
https://arxiv.org/abs/2404.14219
new_line
text
- Gemma 2B: 
- Gemma 2B: 
link
https://arxiv.org/pdf/2403.08295
https://arxiv.org/pdf/2403.08295
# Thoughts on Neural Scaling Laws
When you take a zoomed-out perspective view on the success goals of neural networks, you see they all revolve around the Scaling Laws - empirical observations that performance improves with increased model size, dataset, and compute resources.

The specifics of how these laws apply, vary for different modalities and architectures. This is notable in the empirical equations used to measure these laws. 

Yet they all heavily rely on three main factors - Data, Size and Computation. These factors themselves also have sub-dependencies - data size & quality, model size & architecture, num of GPUs & code for compute kernels respectively.

As research in these laws progresses, we begin to see new scaling laws emerge that may apply in much different ways than usual. This is typical in recent local LLMs (Phi-3, Gemma 2B, LLMs in a flash) which shows small sized models with small rich quality data beating large models

I look forward to the singularity moment - when these laws take a full round spin and meet at where it all began:)

References:
- Scaling Laws for Neural Language Models: https://arxiv.org/pdf/2001.08361
- Scaling Laws for Autoregressive Generative Modeling: https://arxiv.org/abs/2010.14701
- LLMs in a flash: https://arxiv.org/abs/2312.11514
- Phi-3 Technical Report: https://arxiv.org/abs/2404.14219
- Gemma 2B: https://arxiv.org/pdf/2403.08295
👍
ssml2050 ShahabVFX adorkin