Thats a unique idea.You should consider the following idea to enhance your idea so that it would impact uniquely: Emerging paradigms like High Performance Data Analytics (HPDA) and Deep Learning (DL) pose at least two new design challenges for existing MPI runtimes. First, these paradigms require an efficient support for communicating unusually large messages across processes. And second, the communication buffers used by HPDA applications and DL frameworks generally reside on a GPU's memory. In this context,  observe that conventional MPI runtimes have been optimized over decades to achieve lost possible communication latency for relatively smaller message sizes (up-to 1 Megabyte) and that too for CPU memory buffers. With the advent of CUDA-Aware MPI runtimes, a lot of research has been conducted to improve performance of GPU buffer based communication. Hover, little exists in current state of the art that deals with very large message communication of GPU buffers. ,  investigate these new challenges by analyzing the performance bottlenecks in existing CUDA-Aware MPI runtimes like MVAPICH2-GDR, and propose hierarchical collective designs to improve communication latency of the MPI_Bcast primitive by exploiting a new communication library called NCCL. To the best of our knowledge, this is the first work that addresses these new requirements where GPU buffers are used for communication with message sizes surpassing hundreds of megabytes.  highlight the design challenges for our work along with the details of design and implementation. In addition,  provide a comprehensive performance evaluation using a Micro-benchmark and a CUDA-Aware adaptation of Microsoft CNTK DL framework.  report up to 47% improvement in training time for CNTK using the proposed hierarchical MPI_Bcast design.
['Ammar Ahmad Awan', 'Khaled Hamidouche', 'Akshay Venkatesh', 'Dhabaleswar K. Panda']
I am writing a research paper on a idea Efficient Large Message Broadcast using NCCL and CUDA-Aware MPI for Deep Learning
[{"from": "human", "value": "I am writing a research paper on a idea Efficient Large Message Broadcast using NCCL and CUDA-Aware MPI for Deep Learning"}, {"from": "gpt", "value": "Thats a unique idea.You should consider the following idea to enhance your idea so that it would impact uniquely: Emerging paradigms like High Performance Data Analytics (HPDA) and Deep Learning (DL) pose at least two new design challenges for existing MPI runtimes. First, these paradigms require an efficient support for communicating unusually large messages across processes. And second, the communication buffers used by HPDA applications and DL frameworks generally reside on a GPU's memory. In this context,  observe that conventional MPI runtimes have been optimized over decades to achieve lost possible communication latency for relatively smaller message sizes (up-to 1 Megabyte) and that too for CPU memory buffers. With the advent of CUDA-Aware MPI runtimes, a lot of research has been conducted to improve performance of GPU buffer based communication. Hover, little exists in current state of the art that deals with very large message communication of GPU buffers. ,  investigate these new challenges by analyzing the performance bottlenecks in existing CUDA-Aware MPI runtimes like MVAPICH2-GDR, and propose hierarchical collective designs to improve communication latency of the MPI_Bcast primitive by exploiting a new communication library called NCCL. To the best of our knowledge, this is the first work that addresses these new requirements where GPU buffers are used for communication with message sizes surpassing hundreds of megabytes.  highlight the design challenges for our work along with the details of design and implementation. In addition,  provide a comprehensive performance evaluation using a Micro-benchmark and a CUDA-Aware adaptation of Microsoft CNTK DL framework.  report up to 47% improvement in training time for CNTK using the proposed hierarchical MPI_Bcast design."}]