text
𝗔𝗿𝗲 𝘀𝗰𝗮𝗹𝗶𝗻𝗴 𝗹𝗮𝘄𝘀 𝗼𝘃𝗲𝗿? 𝗔 𝗿𝗲𝗽𝗼𝗿𝘁 𝗳𝗿𝗼𝗺 𝘁𝗵𝗲 𝗜𝗻𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻 𝗮𝗻𝗻𝗼𝘂𝗻𝗰𝗲𝗱 𝘁𝗵𝗮𝘁 𝗢𝗽𝗲𝗻𝗔𝗜 𝗶𝘀 𝘀𝗲𝗲𝗶𝗻𝗴 𝗱𝗶𝗺𝗶𝗻𝗶𝘀𝗵𝗶𝗻𝗴 𝗿𝗲𝘁𝘂𝗿𝗻𝘀 𝗳𝗿𝗼𝗺 𝘀𝗰𝗮𝗹𝗶𝗻𝗴 𝘂𝗽 𝘁𝗵𝗲 𝗻𝗲𝘅𝘁 𝗚𝗣𝗧 𝗺𝗼𝗱𝗲𝗹𝘀.
𝗔𝗿𝗲 𝘀𝗰𝗮𝗹𝗶𝗻𝗴 𝗹𝗮𝘄𝘀 𝗼𝘃𝗲𝗿? 𝗔 𝗿𝗲𝗽𝗼𝗿𝘁 𝗳𝗿𝗼𝗺 𝘁𝗵𝗲 𝗜𝗻𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻 𝗮𝗻𝗻𝗼𝘂𝗻𝗰𝗲𝗱 𝘁𝗵𝗮𝘁 𝗢𝗽𝗲𝗻𝗔𝗜 𝗶𝘀 𝘀𝗲𝗲𝗶𝗻𝗴 𝗱𝗶𝗺𝗶𝗻𝗶𝘀𝗵𝗶𝗻𝗴 𝗿𝗲𝘁𝘂𝗿𝗻𝘀 𝗳𝗿𝗼𝗺 𝘀𝗰𝗮𝗹𝗶𝗻𝗴 𝘂𝗽 𝘁𝗵𝗲 𝗻𝗲𝘅𝘁 𝗚𝗣𝗧 𝗺𝗼𝗱𝗲𝗹𝘀.
new_line
new_line
text
📊 What are scaling laws? These are empiric laws that say "Every time you increase compute spent in training 10-fold, your LLM's performance will go up by a predictable tick". Of course, they apply only if you train your model with the right methods.
📊 What are scaling laws? These are empiric laws that say "Every time you increase compute spent in training 10-fold, your LLM's performance will go up by a predictable tick". Of course, they apply only if you train your model with the right methods.
new_line
new_line
text
The image below illustrates it: they're from a paper by Google, "Scaling Autoregressive Models for Content-Rich Text-to-Image Generation", and they show how quality and instruction following of models improve when you scale the model up (which is equivalent to scaling up the compute spent in training).
The image below illustrates it: they're from a paper by Google, "Scaling Autoregressive Models for Content-Rich Text-to-Image Generation", and they show how quality and instruction following of models improve when you scale the model up (which is equivalent to scaling up the compute spent in training).
new_line
new_line
text
➡️ These scaling laws have immense impact: they triggered the largest gold rush ever, with companies pouring billions into scaling up theiur training. Microsoft and OpenAI spent 100B into their "Startgate" mega training cluster, due to start running in 2028.
➡️ These scaling laws have immense impact: they triggered the largest gold rush ever, with companies pouring billions into scaling up theiur training. Microsoft and OpenAI spent 100B into their "Startgate" mega training cluster, due to start running in 2028.
new_line
new_line
text
🤔 So, what about these reports of scaling laws slowing down?
🤔 So, what about these reports of scaling laws slowing down?
new_line
new_line
text
If they are true, they would mean a gigantic paradigm shift, as the hundreds of billions poured by AI companies into scaling could be a dead-end. ⛔️
If they are true, they would mean a gigantic paradigm shift, as the hundreds of billions poured by AI companies into scaling could be a dead-end. ⛔️
new_line
new_line
text
But I doubt it: until the most recent publications, scaling laws showed no signs of weakness, and the researchers at the higher end of the scale-up seems to imply the scaling up continues. 
But I doubt it: until the most recent publications, scaling laws showed no signs of weakness, and the researchers at the higher end of the scale-up seems to imply the scaling up continues. 
new_line
new_line
text
Wait and see!
Wait and see!
𝗔𝗿𝗲 𝘀𝗰𝗮𝗹𝗶𝗻𝗴 𝗹𝗮𝘄𝘀 𝗼𝘃𝗲𝗿? 𝗔 𝗿𝗲𝗽𝗼𝗿𝘁 𝗳𝗿𝗼𝗺 𝘁𝗵𝗲 𝗜𝗻𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻 𝗮𝗻𝗻𝗼𝘂𝗻𝗰𝗲𝗱 𝘁𝗵𝗮𝘁 𝗢𝗽𝗲𝗻𝗔𝗜 𝗶𝘀 𝘀𝗲𝗲𝗶𝗻𝗴 𝗱𝗶𝗺𝗶𝗻𝗶𝘀𝗵𝗶𝗻𝗴 𝗿𝗲𝘁𝘂𝗿𝗻𝘀 𝗳𝗿𝗼𝗺 𝘀𝗰𝗮𝗹𝗶𝗻𝗴 𝘂𝗽 𝘁𝗵𝗲 𝗻𝗲𝘅𝘁 𝗚𝗣𝗧 𝗺𝗼𝗱𝗲𝗹𝘀.

📊 What are scaling laws? These are empiric laws that say "Every time you increase compute spent in training 10-fold, your LLM's performance will go up by a predictable tick". Of course, they apply only if you train your model with the right methods.

The image below illustrates it: they're from a paper by Google, "Scaling Autoregressive Models for Content-Rich Text-to-Image Generation", and they show how quality and instruction following of models improve when you scale the model up (which is equivalent to scaling up the compute spent in training).

➡️ These scaling laws have immense impact: they triggered the largest gold rush ever, with companies pouring billions into scaling up theiur training. Microsoft and OpenAI spent 100B into their "Startgate" mega training cluster, due to start running in 2028.

🤔 So, what about these reports of scaling laws slowing down?

If they are true, they would mean a gigantic paradigm shift, as the hundreds of billions poured by AI companies into scaling could be a dead-end. ⛔️

But I doubt it: until the most recent publications, scaling laws showed no signs of weakness, and the researchers at the higher end of the scale-up seems to imply the scaling up continues. 

Wait and see!
🔥
Smorty100 Viewegger
👀
John6666
/avatars/5cbfa6cbde933503bbc3577cf713e7b5.svg
Friedrich Marty
Smorty100
user