Title: A Main/Subsidiary Network Framework for Simplifying Binary Neural Networks. Abstract: To reduce memory footprint and run-time latency, techniques such as neural net-work pruning and binarization have been explored separately.  However, it is un-clear how to combine the best of the two worlds to get extremely small and efficient models.  In this paper, we, for the first time, define the filter-level pruning problem for binary neural networks, which cannot be solved by simply migrating existing structural pruning methods for full-precision models.  A novel learning-based approach is proposed to prune filters in our main/subsidiary network frame-work, where the main network is responsible for learning representative features to optimize the prediction performance, and the subsidiary component works as a filter selector on the main network. To avoid gradient mismatch when training the subsidiary component, we propose a layer-wise and bottom-up scheme.  We also provide the theoretical and experimental comparison between our learning-based and greedy rule-based methods.  Finally, we empirically demonstrate the effectiveness of our approach applied on several binary models,  including binarizedNIN, VGG-11, and ResNet-18, on various image classification datasets.  For bi-nary ResNet-18 on ImageNet, we use 78.6% filters but can achieve slightly better test error 49.87% (50.02%-0.15%) than the original model