chen-yingfa/CFDBench-raw
Viewer
•
Updated
•
5.13B
•
52
Yingfa Chen
chen-yingfa
AI & ML interests
Long-context modeling, continual learning, architectures
Recent Activity
updated
a dataset
about 1 month ago
chen-yingfa/CFDBench-raw
upvoted
a
paper
2 months ago
MARS: Unleashing the Power of Variance Reduction for Training Large
Models
authored
a paper
3 months ago
Sparsing Law: Towards Large Language Models with Greater Activation
Sparsity
Organizations
None yet
chen-yingfa's activity
updated
a
dataset
about 1 month ago
upvoted
a
paper
2 months ago
authored
2
papers
3 months ago
upvoted
a
paper
3 months ago
commented
a
paper
3 months ago
updated
2
datasets
5 months ago
upvoted
an
article
5 months ago
Article
A failed experiment: Infini-Attention, and why we should keep trying?
•
57
authored
4
papers
5 months ago
CFDBench: A Large-Scale Benchmark for Machine Learning Methods in Fluid Dynamics
Paper
•
2310.05963
•
Published
Robust and Scalable Model Editing for Large Language Models
Paper
•
2403.17431
•
Published
$\infty$Bench: Extending Long Context Evaluation Beyond 100K Tokens
Paper
•
2402.13718
•
Published
•
1
Sub-Character Tokenization for Chinese Pretrained Language Models
Paper
•
2106.00400
•
Published
upvoted
a
paper
7 months ago
authored
a
paper
7 months ago
Can you add some details about this model
2
#1 opened 8 months ago
by
chen-yingfa
Please can you provide a example of use the model weights?
4
#1 opened about 1 year ago
by
Ulov888
liked
a
model
8 months ago
reacted to
akhaliq's
post with ❤️
9 months ago
Post
4413
Leave No Context Behind
Efficient Infinite Context Transformers with Infini-attention
Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention (2404.07143)
This work introduces an efficient method to scale Transformer-based Large Language Models (LLMs) to infinitely long inputs with bounded memory and computation. A key component in our proposed approach is a new attention technique dubbed Infini-attention. The Infini-attention incorporates a compressive memory into the vanilla attention mechanism and builds in both masked local attention and long-term linear attention mechanisms in a single Transformer block. We demonstrate the effectiveness of our approach on long-context language modeling benchmarks, 1M sequence length passkey context block retrieval and 500K length book summarization tasks with 1B and 8B LLMs. Our approach introduces minimal bounded memory parameters and enables fast streaming inference for LLMs.
Efficient Infinite Context Transformers with Infini-attention
Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention (2404.07143)
This work introduces an efficient method to scale Transformer-based Large Language Models (LLMs) to infinitely long inputs with bounded memory and computation. A key component in our proposed approach is a new attention technique dubbed Infini-attention. The Infini-attention incorporates a compressive memory into the vanilla attention mechanism and builds in both masked local attention and long-term linear attention mechanisms in a single Transformer block. We demonstrate the effectiveness of our approach on long-context language modeling benchmarks, 1M sequence length passkey context block retrieval and 500K length book summarization tasks with 1B and 8B LLMs. Our approach introduces minimal bounded memory parameters and enables fast streaming inference for LLMs.
liked
a
dataset
9 months ago