Network with Sub-networks: Layer-wise Detachable Neural Network
We introduce network with sub-networks, a neural network which it’s weight layers could be detached into subneural networks during inference. To develop weights and biases which could be inserted in both base and subneural networks, firstly, the parameters are copied from sub-model to base-model. Each model is forwardpropagated separately. Gradients from a pair of networks are averaged and, used to update both networks. Our base model achieves the test-accuracy which is comparable to the regularly trained models, while the model maintains the ability to detach weight layers.