@inproceedings{zhao-etal-2023-tencentpretrain,
    title = "{T}encent{P}retrain: A Scalable and Flexible Toolkit for Pre-training Models of Different Modalities",
    author = "Zhao, Zhe  and
      Li, Yudong  and
      Hou, Cheng  and
      Zhao, Jing  and
      Tian, Rong  and
      Liu, Weijie  and
      Chen, Yiren  and
      Sun, Ningyuan  and
      Liu, Haoyan  and
      Mao, Weiquan  and
      Guo, Han  and
      Gou, Weigang  and
      Wu, Taiqiang  and
      Zhu, Tao  and
      Shi, Wenhang  and
      Chen, Chen  and
      Huang, Shan  and
      Chen, Sihong  and
      Liu, Liqun  and
      Li, Feifei  and
      Chen, Xiaoshuai  and
      Sun, Xingwu  and
      Kang, Zhanhui  and
      Du, Xiaoyong  and
      Shen, Linlin  and
      Yan, Kimmo",
    editor = "Bollegala, Danushka  and
      Huang, Ruihong  and
      Ritter, Alan",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-demo.20",
    doi = "10.18653/v1/2023.acl-demo.20",
    pages = "217--225",
    abstract = "Recently, the success of pre-training in text domain has been fully extended to vision, audio, and cross-modal scenarios. The proposed pre-training models of different modalities are showing a rising trend of homogeneity in their model structures, which brings the opportunity to implement different pre-training models within a uniform framework. In this paper, we present TencentPretrain, a toolkit supporting pre-training models of different modalities. The core feature of TencentPretrain is the modular design. The toolkit uniformly divides pre-training models into 5 components: embedding, encoder, target embedding, decoder, and target. As almost all of common modules are provided in each component, users can choose the desired modules from different components to build a complete pre-training model. The modular design enables users to efficiently reproduce existing pre-training models or build brand-new one. We test the toolkit on text, vision, and audio benchmarks and show that it can match the performance of the original implementations.",
}

Recently, the success of pre-training in text domain has been fully extended to vision, audio, and cross-modal scenarios. The proposed pre-training models of different modalities are showing a rising trend of homogeneity in their model structures, which brings the opportunity to implement different pre-training models within a uniform framework. In this paper, we present TencentPretrain, a toolkit supporting pre-training models of different modalities. The core feature of TencentPretrain is the modular design. The toolkit uniformly divides pre-training models into 5 components: embedding, encoder, target embedding, decoder, and target. As almost all of common modules are provided in each component, users can choose the desired modules from different components to build a complete pre-training model. The modular design enables users to efficiently reproduce existing pre-training models or build brand-new one. We test the toolkit on text, vision, and audio benchmarks and show that it can match the performance of the original implementations.
Zhao, Zhe Li, Yudong Hou, Cheng Zhao, Jing Tian, Rong Liu, Weijie Chen, Yiren Sun, Ningyuan Liu, Haoyan Mao, Weiquan Guo, Han Gou, Weigang Wu, Taiqiang Zhu, Tao Shi, Wenhang Chen, Chen Huang, Shan Chen, Sihong Liu, Liqun Li, Feifei Chen, Xiaoshuai Sun, Xingwu Kang, Zhanhui Du, Xiaoyong Shen, Linlin Yan, Kimmo
TencentPretrain: A Scalable and Flexible Toolkit for Pre-training Models of Different Modalities