text
Web Rephrase Augmented Pre-training (WRAP) enhances language model training efficiency by transforming documents into structured formats.
Web Rephrase Augmented Pre-training (WRAP) enhances language model training efficiency by transforming documents into structured formats.
new_line
new_line
text
Key aspects:
Key aspects:
new_line
text
* Utilizes an instruction-tuned model to rephrase web content into styles such as Wikipedia or Q/A, creating a blend of synthetic and real data for training.
* Utilizes an instruction-tuned model to rephrase web content into styles such as Wikipedia or Q/A, creating a blend of synthetic and real data for training.
new_line
text
* Demonstrated improvements of over 10% better perplexity, alongside more than 2% increase in zero-shot question-answering accuracy.
* Demonstrated improvements of over 10% better perplexity, alongside more than 2% increase in zero-shot question-answering accuracy.
new_line
new_line
text
Congrats to the authors for their work!
Congrats to the authors for their work!
new_line
new_line
text
Paper: 
Paper: 
resource
https://huggingface.co/papers/2401.16380
paper
2401.16380
https://huggingface.co/papers/2401.16380
Rephrasing the Web: A Recipe for Compute and Data-Efficient Language
  Modeling (2401.16380)
Web Rephrase Augmented Pre-training (WRAP) enhances language model training efficiency by transforming documents into structured formats.

Key aspects:
* Utilizes an instruction-tuned model to rephrase web content into styles such as Wikipedia or Q/A, creating a blend of synthetic and real data for training.
* Demonstrated improvements of over 10% better perplexity, alongside more than 2% increase in zero-shot question-answering accuracy.

Congrats to the authors for their work!

Paper: https://huggingface.co/papers/2401.16380
❤️
osanseviero clem lunarflu BroCodeR Alexandro14 kotyKD abdullah
https://cdn-avatars.huggingface.co/v1/production/uploads/657217faabb25ed8aedd5e48/UUHAXeGtOnQBXFD3nYtf2.jpeg
Vlad Bogolin
vladbogo
user