CSTAR: Towards Compact and STructured Deep Neural Networks with Adversarial Robustness

Model compression and model defense for deep neural net- works (DNNs) have been extensively and individually studied. Considering the co-importance of model compactness and robustness in practical applications, several prior works have explored to improve the adversarial robustness of the sparse neural networks. However, the structured sparse models obtained by the existing works suffer severe performance degradation for both benign and robust accuracy, thereby causing a challenging dilemma between robustness and struc- turedness of compact DNNs. To address this problem, in this paper, we propose CSTAR, an efﬁcient solution that simultaneously impose C ompactness, high ST ructuredness and high A dversarial R obustness on the target DNN models. By formulating the structuredness and robustness requirement within the same framework, the com- pressed DNNs can simultaneously achieve high compression performance and strong adversarial robustness. Evaluations for various DNN models on different datasets demonstrate the effectiveness of CSTAR. Compared with the state-of-the-art robust structured pruning, CSTAR shows consistently better performance. For instance, when compressing ResNet-18 on CIFAR-10, CSTAR achieves up to 20 . 07% and 11 . 91% improvement for benign accuracy and robust accuracy, respec- tively. For compressing ResNet-18 with 16 × compression ra- tio on Imagenet, CSTAR obtains 8 . 58% benign accuracy gain and 4 . 27% robust accuracy gain compared to the existing robust structured pruning.