arXiv:1711.07459v1 [] 20 Nov 2017

SquishedNets: Squishing SqueezeNet further for edge
device scenarios via deep evolutionary synthesis

Mohammad Javad Shafiee
Dept. of Systems Design Engineering
University of Waterloo, DarwinAI
mjshafiee@uwaterloo.ca

Francis Li
Dept. of Systems Design Engineering
University of Waterloo, DarwinAI
francis@darwinai.ca

Brendan Chwyl
Dept. of Systems Design Engineering
University of Waterloo, DarwinAI
brendan@darwinai.ca

Alexander Wong
Dept. of Systems Design Engineering
University of Waterloo, DarwinAI
a28wong@uwaterloo.ca

Abstract
While deep neural networks have been shown in recent years to outperform other
machine learning methods in a wide range of applications, one of the biggest challenges with enabling deep neural networks for widespread deployment on edge devices such as mobile and other consumer devices is high computational and memory requirements. Recently, there has been greater exploration into small deep
neural network architectures that are more suitable for edge devices, with one of
the most popular architectures being SqueezeNet, with an incredibly small model
size of 4.8MB. Taking further advantage of the notion that many applications of
machine learning on edge devices are often characterized by a low number of target classes, this study explores the utility of combining architectural modifications
and an evolutionary synthesis strategy for synthesizing even smaller deep neural
architectures based on the more recent SqueezeNet v1.1 macroarchitecture (considered state-of-the-art in efficient architectures) for applications with fewer target classes. In particular, architectural modifications are first made to SqueezeNet
v1.1 to accommodate for a 10-class ImageNet-10 dataset, and then an evolutionary
synthesis strategy is leveraged to synthesize more efficient deep neural networks
based on this modified macroarchitecture. The resulting SquishedNets possess
model sizes ranging from 2.4MB to 0.95MB (∼5.17X smaller than SqueezeNet
v1.1, or 253X smaller than AlexNet). Furthermore, the SquishedNets are still able
to achieve accuracies ranging from 81.2% to 77%, and able to process at speeds of
156 images/sec to as much as 256 images/sec on a Nvidia Jetson TX1 embedded
chip. These preliminary results show that a combination of architectural modifications and an evolutionary synthesis strategy can be a useful tool for producing
very small deep neural network architectures that are well-suited for edge device
scenarios without the need for compression or quantization.

1 Introduction
One can consider deep neural networks [5] to be one of the most successful machine learning methods, outperforming many state-of-the-art machine learning methods in a wide range of applications
ranging from image categorization [4] to speech recognition. A very major factor to the great recent successes of deep neural networks has been the availability of very powerful high performance
computing systems due to the great advances in parallel computing hardware. This has enabled
31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.

researchers to significantly increase the depth and complexity of deep neural networks, resulting in
greatly increased modeling capabilities. As such, the majority of research in deep neural networks
have largely focused on designing deeper and more complex deep neural network architectures for
improved accuracy. However, the increasing complexity of deep neural networks has become one
of the biggest obstacles to the widespread deployment of deep neural networks on edge devices
such as mobile and other consumer devices, where computational, memory, and power capacity is
significantly lower than that in high performance computing systems.
Given the proliferation of edge devices and the increasing demand for machine learning applications in such devices, there has been an increasing amount of research exploration on the design of
smaller, more efficient deep neural network architectures that can both infer and train faster, as well
as transfer faster onto embedded chips that power such edge devices. One commonly employed approach for designing smaller neural network architectures is synaptic precision reduction, where the
number of bits used to represent synaptic strength is significantly reduced from 32-bit floating point
precision to fixed-point precision [12], 2-bit precision [6, 13, 7], or 1-bit precision [1, 8]. While this
approach leads to greatly reduced model sizes, the resulting deep neural networks often require specialized hardware support for accelerated deep inference and training on embedded devices, which
can limit their utility for wide range of applications.
Another approach to designing smaller deep neural network architectures is to take a principled
approach and employ architectural design strategies to achieve more efficient deep neural network
macroarchitectures [2, 3]. An exemplary case of what can be achieved using such an approach
is SqueezeNet [3], where three key design strategies where employed: 1) decrease the number of
3x3 filters, 2) decrease the number of input channels to 3x3 filters, and 3) downsample late in
the network. This resulted in a macroarchitecture composed of Fire modules that possessed an
incredibly small model size of 4.8MB, which is 50X smaller than AlexNet with comparable accuracy
on ImageNet for 1000 classes. The authors further introduced SqueezeNet v1.1, where the number
of filters as well as the filter sizes are further reduced, resulting in 2.4X less computation than
the original SqueezeNet without sacrificing accuracy and thus can be considered state-of-the-art in
efficient network architectures.
Inspired by the incredibly small macroarchitecture of SqueezeNet, we are motivated to take it one
step further by taking into account that the majority of applications of machine learning on edge
devices such as mobile and consumer devices are quite specialized and often require a much fewer
number of target classes (typically less than 10 target classes). As such, this study explores the utility of combining architectural modifications and an evolutionary synthesis strategy for synthesizing
even smaller deep neural architectures based on the SqueezeNet v1.1 macroarchitecture for applications with fewer target classes. We will refer to these smaller deep neural network architectures as
SquishedNets.

2 Architectural modification for fewer target classes
The first and simplest strategy taken in this study is to perform some simple architectural modifications to the macroarchitecture of SqueezeNet v1.1 to accommodate for scenarios where we are
dealing with much fewer target classes. For this study, we explored a classification scenario where
the number of target classes is reduced to 10, as fewer target classes is quite common for many machine learning applications on edge devices such as mobile and other consumer devices. Given the
reduced number of target classes, we modify the conv10 layer to a set of 10 1x1 filters. Given that
the conv10 layer contains ∼40% of all parameters in SqueezeNet [3], this architectural modification
resulted in a significant reduction in model size. While not a leap of the imagination and a rather
trivial modification, this illustrates that SqueezeNet v1.1 is a great macroarchitecture that can be
modified to be even more efficient for edge scenarios where there are fewer target classes.

3 Evolutionary synthesis of more efficient network architectures
The second strategy taken in this study is to employ an evolutionary synthesis strategy [10, 11, 9] to
synthesize even more efficient deep neural network architectures than can be achieved through principled macroarchitecture design strategies. First proposed by [10] and subsequently extended [11, 9],
the idea behind evolutionary synthesis is to automatically synthesize progressively more efficient
deep neural networks over successive generations in a stochastic synaptogenesis manner within a
probabilistic framework. More specifically, synaptic probability models are used to encode the ge2

Table 1: ImageNet-10 dataset
wnid
n02783161
n03085013
n04557648
n04037443
n03793489

Class Name
pen
keyboard
water bottle
car
computer mouse

wnid
n03584254
n04548362
n07930864
n03782006
n04409515

Class Name
cell phone
wallet
cup
monitor
tennis ball

netic information of a deep neural network, thus mimic the notion of heredity. Offspring networks
are then synthesized in a stochastic manner given the synaptic probability models and a set of computational environmental constraints for influencing synaptogenesis, thus forming the next generation
of deep neural networks, thus mimicking the notions of random mutation and natural selection. The
offspring deep neural networks are then trained and this evolution synthesis process is performed
over generations until the desired traits are met.
While a more detailed description of the evolutionary synthesis strategy can be found in [10], a
brief mathematical description is provided as follows. Let the genetic encoding of a network be
formulated as P (Hg |Hg−1 ), where the network architecture Hg at generation g is influenced by the
network architecture Hg−1 of generation g − 1. An offspring network is synthesized in a stochastic
manner via a synthesis probability P (Hg ), which combines the genetic encoding P (Hg |Hg−1 ) with
the environmental factor model R being imposed:
P (Hg ) ≈ P (Hg |Hg−1 ) · R.

(1)

Since the underlying goal is to influence the synaptogenesis behaviour of offspring deep neural networks to be progressively more efficient generation after generation, the environmental factor model
R is set to a value less than 1 to further enforce a resource-starved environment on the deep neural
networks from generation to generation. In this study, the aforementioned modified macroarchitecture is used as the ancestral precursor for the evolutionary synthesis process to produce deep neural
networks with even smaller network architectures. In this study, 15 generations of evolutionary
synthesis was performed to produce the final SquishedNets.
Table 2: Performance results of SquishedNets.
Model
Name

Model
size

SquishedNet-1
SquishedNet-2
SquishedNet-3
SquishedNet-4

2.4MB
2.0MB
1.3MB
0.95MB

Reduction in
model size
vs. SqueezeNet v1.1
2.04X
2.45X
3.77X
5.17X

Reduction in
model size
vs. AlexNet
100X
120X
184X
253X

Runtime
speed
(images/sec)
156.09
174.86
225.35
256.00

Top-1
accuracy
(ImageNet-10)
81.2%
79.6%
78.6%
77.0%

4 Preliminary Results and Discussion
To study the utility of a combination of architectural modifications and evolutionary synthesis on
synthesizing very small deep neural network architectures based on the SqueezeNet v1.1 macroarchitecture that are well-suited for edge device scenarios, we examine the top-1 accuracies and runtime speeds (on an Nvidia Jetson TX1 embedded chip with batch size of 32) of our synthesized
SquishedNets on the 10-class ImageNet-10 dataset. The ImageNet-10 dataset used in this study is
a subset of the ImageNet dataset composed of the following ten target classes reported in Table 1.
The performance results of four different SquishedNets (produced at four different generations of
the evolutionary synthesis process) are shown in Table 1. A number of observations can be made
based on Table 2. First, it can be observed that leveraging both architectural modifications to account
for fewer target classes as well as evolutionary synthesis results in the generation of even more efficient network architectures, as evident by the SquishedNets having model sizes range from 2.4MB
to just 0.95MB. Therefore, the smallest SquishedNet is 5.17X smaller than SqueezeNet v1.1 (or
∼253X smaller than AlexNet). Second, not only was there a significant model size reductions, the
SquishedNets were able to process at speeds of 156 images/sec to as much as 256 images/sec on a
3

Nvidia Jetson TX1 embedded chip, which is significant particularly for edge scenarios with mobile
and other consumer devices. Therefore, the ability to not only achieve very small model sizes but
also fast runtime speeds has great benefits when used in resource-starved environments with limited
computational, memory, and energy requirements. In terms of top-1 accuracy, the SquishedNets are
able to still achieve accuracies ranging from 81.2% to 77.0%, which is high enough for many edge
applications. These preliminary results show that a combination of architectural modifications and
an evolutionary synthesis strategy can be a useful tool for producing very small deep neural network architectures that are well-suited for edge device scenarios without the need for compression
or quantization.

Acknowledgment
The authors thank NSERC, the Canada Research Chairs program, Nvidia, and DarwinAI.

References
[1] M. Courbariaux, Y. Bengio, and J.-P. David. Binaryconnect: Training deep neural networks with binary
weights during propagations. NIPS, 2015.
[2] A. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam.
Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861,
2017.
[3] Forrest N. Iandola, Song Han, Matthew W. Moskewicz, Khalid Ashraf, William J. Dally, and Kurt
Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and <0.5mb model size.
arXiv:1602.07360, 2016.
[4] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional
neural networks. In NIPS, 2012.
[5] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 2015.
[6] Fengfu Li, Bo Zhang, and Bin Liu. Ternary weight networks. arXiv:1605.04711, 2016.
[7] W. Meng, Z. Gu, M. Zhang, and Z. Wu. Two-bit networks for deep learning on resource-constrained
embedded devices. arXiv:1701.00485, 2017.
[8] M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi. Xnor-net: Imagenet classification using binary
convolutional neural networks. ECCV, 2015.
[9] M. Shafiee, F. Li, and A. Wong. Exploring the imposition of synaptic precision restrictions for evolutionary synthesis of deep neural networks. In https://arxiv.org/abs/1707.00095, 2016.
[10] M. Shafiee, A. Mishra, and A. Wong. Deep learning with darwin: Evolutionary synthesis of deep neural
networks. arXiv:1606.04393, 2016.
[11] M. Shafiee and A. Wong. Evolutionary synthesis of deep neural networks via synaptic cluster-driven
genetic encoding. In NIPS Workshop, 2016.
[12] S. Shin, Y. Boo, and W. Sung. Fixed-point optimization of deep neural networks with adaptive step size
retraining. arXiv:1702.08171, 2017.
[13] P. Yin, S. Zhang, J. Xin, and Y. Qi. Training ternary neural networks with exact proximal operator.
arXiv:1612.06052, 2017.

4

