- Hao Zhang, Yuan Li, Zhijie Deng, Xiaodan Liang, L. Carin, E. Xing. 2020. AutoSync: Learning to Synchronize for Data-Parallel Distributed Deep Learning. Abstract: The rationale behind Eq. 1 is as follows: (1) Since many runtime systems (e.g. TensorFlow [1] or PyTorch [3]) introduce scheduling or parallelization between communication and computation, in practice, there are significant overlaps between the two components; (2) in data-parallel training, it is commonly observed that one component usually dominates the other [4]. These make using the maximum of them as the estimation of the total time reasonable.
