text
nanoGPT with Sigmoid Self-Attention
nanoGPT with Sigmoid Self-Attention
new_line
text
I couldn’t resist had to give it a try:)
I couldn’t resist had to give it a try:)
new_line
new_line
text
Some observations on M2:
Some observations on M2:
new_line
text
SSA was ~5-10% faster in training with similar final loss values, slightly less coherent text generation, marginally higher perplexity, and lower memory usage compared to softmax.
SSA was ~5-10% faster in training with similar final loss values, slightly less coherent text generation, marginally higher perplexity, and lower memory usage compared to softmax.
new_line
new_line
text
Code: 
Code: 
link
https://github.com/Jaykef/ai-algorithms/blob/main/sigmoid_attn.ipynb
https://github.com/Jaykef/ai-algorithms/blob/main/sigmoid_attn.ipynb
nanoGPT with Sigmoid Self-Attention
I couldn’t resist had to give it a try:)

Some observations on M2:
SSA was ~5-10% faster in training with similar final loss values, slightly less coherent text generation, marginally higher perplexity, and lower memory usage compared to softmax.

Code: https://github.com/Jaykef/ai-algorithms/blob/main/sigmoid_attn.ipynb
👀
John6666 prince-canuma KingNish louisbrulenaudet
🔥
prince-canuma mtasic85