arxiv:2412.19638

Xmodel-2 Technical Report

Published on Dec 27, 2024

· Submitted by

valeriaWong on Jan 2

#2 Paper of the day

Upvote

Authors:

Wang Qun ,

Abstract

Xmodel-2 is a 1.2-billion-parameter large language model designed specifically for reasoning tasks. Its architecture enables different model scales to share a unified set of hyperparameters, allowing for extensive experimentation on smaller models and seamless transfer of optimal configurations to larger models. To maximize training efficiency and stability, Xmodel-2 employs the WSD learning rate scheduler from MiniCPM. Pretrained on 1.5 trillion tokens from diverse sources, Xmodel-2 achieves state-of-the-art performance in complex reasoning and agent-based tasks, while maintaining low training costs. These results highlight the potential of efficient model design and training strategies in advancing reasoning capabilities. Model checkpoints and code are publicly available on GitHub at https://github.com/XiaoduoAILab/Xmodel-2

View arXiv page View PDF Add to collection

Community

valeriaWong

Paper author Paper submitter 4 days ago

•

edited 4 days ago

This paper is a significant contribution to the field, showcasing a remarkable achievement in complex reasoning tasks using models in the 1–2B parameter range. The authors achieved SOTA-level performance while utilizing only 1.5T tokens, a stark contrast to the 18T tokens required by comparable models. This efficiency is driven by their innovative WSD learner strategy and the integration of tensor program methods.
Furthermore, the paper provides a groundbreaking analysis of the SFT data ratio in the decay phase of the WSD learner strategy, addressing a key area often overlooked in previous research. The research's practical implications are underscored by the authors' commitment to open-sourcing both the model and the code, enabling broader accessibility and fostering further advancements in the field.