Title: Variability of Neural Networks and Han-Layer: A Variability-Inspired Model. Abstract: What makes an artificial neural network easier to train or to generalize better than its peers?  We introduce a notion of variability to view such issues under the setting of a fixed number of parameters which is, in general, a dominant cost-factor.  Experiments verify that variability correlates positively to the number of activations and negatively to a phenomenon called Collapse to Constants, which is related but not identical to vanishing gradient.  Further experiments on stylized problems show that variability is indeed a key performance indicator for fully-connected neural networks.  Guided by variability considerations, we propose a new architecture called Householder-absolute neural layers, or Han-layers for short, to build high variability networks with a guaranteed immunity to gradient vanishing or exploding.
On small stylized models, Han-layer networks exhibit a far superior generalization ability over fully-connected networks.  Extensive empirical results demonstrate that, by judiciously replacing fully-connected layers in large-scale networks such as MLP-Mixers, Han-layers can greatly reduce the number of model parameters while maintaining or improving generalization performance.  We will also briefly discuss current limitations of the proposed Han-layer architecture.
