Weronika Stryj
privategeek24
AI & ML interests
Text classification, Sentiment Analysis, Finance Risk analysis, Text generation, Question answearing, Optimization LLM, Cyber risk analysis, Speech to text
Recent Activity
liked
a Space
about 1 month ago
sentence-transformers/quantized-retrieval
reacted
to
singhsidhukuldeep's
post
with 👀
3 months ago
While Google's Transformer might have introduced "Attention is all you need," Microsoft and Tsinghua University are here with the DIFF Transformer, stating, "Sparse-Attention is all you need."
The DIFF Transformer outperforms traditional Transformers in scaling properties, requiring only about 65% of the model size or training tokens to achieve comparable performance.
The secret sauce? A differential attention mechanism that amplifies focus on relevant context while canceling out noise, leading to sparser and more effective attention patterns.
How?
- It uses two separate softmax attention maps and subtracts them.
- It employs a learnable scalar λ for balancing the attention maps.
- It implements GroupNorm for each attention head independently.
- It is compatible with FlashAttention for efficient computation.
What do you get?
- Superior long-context modeling (up to 64K tokens).
- Enhanced key information retrieval.
- Reduced hallucination in question-answering and summarization tasks.
- More robust in-context learning, less affected by prompt order.
- Mitigation of activation outliers, opening doors for efficient quantization.
Extensive experiments show DIFF Transformer's advantages across various tasks and model sizes, from 830M to 13.1B parameters.
This innovative architecture could be a game-changer for the next generation of LLMs. What are your thoughts on DIFF Transformer's potential impact?
reacted
to
davidberenstein1957's
post
with 👍
3 months ago
Don't use an LLM when you can use a much cheaper model.
The problem is that no one tells you how to actually do it.
Just picking a pre-trained model (e.g., BERT) and throwing it at your problem won't work!
If you want a small model to perform well on your problem, you need to fine-tune it.
And to fine-tune it, you need data.
The good news is that you don't need a lot of data but instead high-quality data for your specific problem.
In the latest livestream, I showed you guys how to get started with Argilla on the Hub! Hope to see you at the next one.
https://www.youtube.com/watch?v=BEe7shiG3rY
Organizations
None yet
models
None public yet
datasets
None public yet