Transforming Large-size to Lightweight Deep Neural Networks for IoT Applications: A Review

Deep Neural Networks (DNN) have gained unprecedented popularity due to their high-order performance and automated feature extraction capability. It encourages researchers to incorporate DNN in different Internet of Things (IoT) applications in recent years. However, the colossal requirement of computation, energy, and storage of DNN make their deployment prohibitive on resource constraint IoT devices. Therefore, several compression techniques have been proposed in recent years to reduce the energy, storage, and computation requirements of the DNN. These techniques have utilized a different perspective for compressing DNN with minimal accuracy compromise. It encourages us to comprehensively overview the DNN compression techniques for the IoT. This paper presents a comprehensive overview of existing literature on compressing the DNN that reduces energy consumption, storage, and computation requirements for IoT applications. We divide the existing approaches into five broad categories, i.e., network pruning, sparse representation, bits precision, knowledge distillation, and miscellaneous, based upon the mechanism incorporated for compressing the DNN. The paper discusses the challenges associated with each category of DNN compression techniques and presents some prominent applications using IoT in conjunction with compressed DNN. Finally, we provide a quick summary of existing work under each category with the future direction in DNN compression.