- Jinliang Wei, Aurick Qiao, Anand Jayarajan, Garth A. Gibson, Vijay Vasudevan, E. Xing. 2019. Training Larger Models on TensorFlow without Additional GPU. Abstract: Across different applications, we have seen examples that models with larger capacity achieve better performance. However, the model capacity is ultimately restricted by the limited and expensive GPU memory. In this paper, we present a number of techniques that leverage cheap host memory to reduce GPU memory consumption during training in TensorFlow. Our techniques require no modifications to TensorFlow application programs and enable training models with 3.8× more (up to 2.5 Billion parameters) parameters on a single GPU.
