arXiv:1801.06434v5 [] 14 Mar 2018

EffNet: AN EFFICIENT STRUCTURE FOR CONVOLUTIONAL NEURAL NETWORKS
Ido Freeman, Lutz Roese-Koerner

Anton Kummert

Aptiv
Wuppertal, Germany

University of Wuppertal
Department of Electical Engineering

{first_name.last_name}@aptiv.com

kummert@uni-wuppertal.de

ABSTRACT
With the ever increasing application of Convolutional Neural
Networks to costumer products the need emerges for models which can efficiently run on embedded, mobile hardware.
Slimmer models have therefore become a hot research topic
with multiple different approaches which vary from binary
networks to revised convolution layers. We offer our contribution to the latter and propose a novel convolution block which
significantly reduces the computational burden while surpassing the current state-of-the-art. Our model, dubbed EffNet,
is optimised for models which are slim to begin with and is
created to tackle issues in existing models such as MobileNet
and ShuffleNet.

methods they leave two large issues unaddressed. First, both
papers report taking large networks and making them smaller
and more efficient. When applying their models to slimmer
networks the results start to diverge. Second, both proposed
models create an aggressive bottleneck [7] for data flow in the
network. This kind of a bottleneck might prove insignificant
in models of high redundancy yet, as our experiments show, it
has a destructive effect on smaller models.
We therefore propose an alternative constellation which retains most of the proportional decrease in computations while
having little to no affect on the accuracies. We achieve this
improvement by optimising data flow through the network
and neglecting practices which prove harmful in this unique
domain.

Index Terms— convolutional neural networks, computational efficiency, real-time inference
2. RELATED WORK
1. INTRODUCTION
With recent industrial recognition of the benefits of Artificial
Neural Networks to product capabilities, the demand emerges
for efficient algorithms to run in real-time on cost-effective
hardware. This contradicts, in a way, the now parallel university research. While the latter enjoys a relative freedom
in terms of execution cycles and hardware, the former is subjected to market forces and product requirements.
Over the years multiple papers proposed different approaches for tackling the problem of real-time inference on a
small hardware. One common method is the pruning of trained
networks [1], [2], [3]. Another is the fix-point conversion of
32bit networks to as far as binary models [4]. A more recent
approach concentrates on the interconnectivity of the neurons
and the very nature of the vanilla convolution layers.
A vanilla convolution layer consists, in its core, of a fourdimensional tensor which is swiped over an input signal in
the following format [rows, columns, channels in, channels
out], resulting in a quadruple-component multiplication. This
means that the computational cost scales by a four-fold factor.
As 3 × 3 convolutions are now a standard, they become a
natural candidate for optimisation. Papers as [5] (MobileNet)
and [6] (ShuffleNet) set to solve this issue by separating the
computations along the different dimensions. Yet in their

Much of the work in the field focuses on hyper-parameter
optimisation. Algorithms from this class are rather general
both in terms of target algorithm and optimisation objective.
[8] proposed a general Bayesian optimisation framework for
black-box algorithms as CNNs 1 and SVMs 2 by maximising
the probability of increasing the model’s accuracy. This could
be combined with multi-objective optimisation as in [9], to also
optimise the computational complexity. These methods mostly
work well when initialised properly and many of them are limited in their search space [10]. using reinforcement learning,
[11] trained an LSTM 3 [12] to optimise hyper-parameters for
improved accuracy and speed. This along with recent evolutionary methods [13] exhibits less limitations on the search
space but complicates the development by deploying additional
modules.
An additional approach resolves to decreasing the size of
large models in a post-processing manner. Papers such as
[1], [2] and [3] proposed pruning algorithms with a minimal
cost to accuracies. However pruning leads to several issues.
The pipeline itself requires an additional phase with dedicated
hyper-parameters which require optimisation. Furthermore, as
1 Convolutional

Neural Networks
Vector Machines
3 Long Short-Term Memory
2 Support

the network’s architecture is changed, the models require an
additional fine-tuning step.
A further method for compression through post-processing
is the fix-point quantisation of models to smaller primitives
than the common 32bit floats [14] [15], [16] and the binary networks of [4]. Quantised models, although much faster, consistently show decreased accuracies compared to their baselines
and are thus less appealing.
Last, similar to this work, papers as [17], [5] and [6] revisited the very nature of the common convolution operator.
This involves the dimension-wise separation of the convolution
operator, as discussed in [18]. Here, the original operation is
approximated using significantly less FLOPs. [7] separated
the 3 × 3 kernels into two consecutive kernels of shapes 3 × 1
and 1 × 3. The MobileNet model [5] took a step further and
separated the channel-wise from the spatial convolution which
is also only applied depthwise, see Figure 1b. By doing so
a significant reduction in FLOPs was achieved while the majority of computations was shifted to the point-wise layers.
Finally, the ShuffleNet model [6] addressed the stowage of
FLOPs in the point-wise layers by dividing them into groups
in a similar way to [19]. This lead to a drastic reduction in
FLOPs with a rather small toll on the accuracies, see Figure 1
in [6] and Figure 1c.
The diversity of methods shows that there are multiple
ways to successfully compress a CNN. Yet most methods
assume a large development model which is adjusted for efficiency. They thus commonly seem to reach their limits when
applied to networks which are slim to begin with. As many
embedded systems have a limited specification, models are
normally designed within these limitations rather than just
optimising a large network. In such environments the limitations of [5] and [6] become clearer thus laying the base for
our EffNet model which shows the same capacity even when
applied to shallow and narrow models.
Finally notice that the methods above are not mutually exclusive. For example, our model could also be converted to fixpoint, pruned and optimised for best set of hyper-parameters.

(a) An EffNet block (b) A MobileNet (c) A ShuffleNet
(ours)
block
block

Fig. 1: A comparison of MobileNet and ShuffleNet with our EffNet
blocks. ’dw’ means depthwise convolution, ’mp’ means max-pooling,
’ch’ is for the number of output channels and ’gc’ is for group convolutions. Best seen in colour.
algorithms which are optimised to run a specific task extremely
quick, e.g. [20]. Additionally, regulatory limitations often
prohibit a one network solution as they require backup systems
and highly interpretable decision making processes. Reducing
the computational cost of all small classifiers in a project would
thus allow either the redistribution of computational power to
more critical places or enable deeper and wider models with a
larger capacity.
Exploring the limitations of previous work revealed that the
smaller the model is, the more accuracy it loses when converted
to MobileNet or to ShuffleNet, see section 5. Analysing the
nature of these suggested modifications we came across several
issues.
The Bottleneck Structure The bottleneck structure as discussed in [21] applies a reduction factor of eight to the number
of input channels in a block, comparing to the number of output
channels. A ShuffleNet block uses a reduction factor of four
[6]. Yet narrow layers do not tend to have enough channels
to enable such a drastic reduction. In all of our experiments
we witnessed a loss in accuracy comparing to more moderate
reduction and we therefore propose to use a bottleneck factor
of two. Additionally it was found useful to use the special
convolution (see following paragraph) with a depth multiplier
of 2, i.e. the first depthwise convolution layer also doubles the
amount of channels.
Separable Convolutions Proposed by [7] and used neither
by ShuffleNet nor by MobileNet, we revisit the idea of consecutive separable spatial convolutions, i.e. using 3 × 1 and 1 × 3
layers instead of a single 3 × 3 layer. Separating the spatial
convolution might only make a minor difference in terms of
FLOPs but we could not find any reason not to use it.
Strides and Pooling Both MobileNet and ShuffleNet models choose to apply a stride of two to the depthwise spatial
convolution layer in their blocks. Our experiments show two
issues with this practice. First, we repeatedly witnessed a decrease in accuracy comparing to max-pooling. This was in a
way expected as strided convolution is prone to aliasing.

3. BUILDING BLOCKS FOR INCREASED
EFFICIENCY
This section discusses the most common practices for increasing efficiency. The presented results assisted with identifying
weaknesses in previous techniques and constructing a suitable
solution in the form of a unified EffNet block. For practical
reasons we avoid going into details regarding the exact settings
of the following experiments. Instead we discuss their results
and show their effect as a whole in section 5.
The combination of multiple tasks, competitive costs and
interactive run-times puts strict limitations on possible model
sizes for industrial applications. These requirements in fact
often lead to the usage of more classical computer vision
2

Table 1: Data flow analysis of selected models. One could intuitively understand how an aggressive data compression in early stages would
harm accuracies. Compression factors of 4 or more are marked in red. gc4 means convolution in 4 groups. Best seen in colour.
Baseline

MobileNet [5]

3x3x64 + mp

Floats
Out
16384

3x3x128 + mp

ShuffleNet [6]

3x3x64 + mp

Floats
Out
16384

8192

dw 3x3 + stride
1x1x128

3x3x256 + mp

4096

Fully Connected

10

Layer

EffNet (Ours)

3x3x64 + mp

Floats
Out
16384

4096
8192

gc4 1x1x32
dw 3x3 + stride
gc4 1x1x128

8192
2048
8192

dw 3x3 + stride
1x1x256

2048
4096

gc4 1x1x64
dw 3x3 + stride
gc4 1x1x256

4096
1024
4096

Fully Connected

10

Fully Connected

10

Layer

Layer

Layer
1x1x32
dw 1x3 + 1d mp
dw 3x1
2x1x64 + 1d stride
1x1x64
dw 1x3 + 1d mp
dw 3x1
2x1x128 + 1d stride
1x1x128
dw 1x3 + 1d mp
dw 3x1
2x1x256 + 1d stride
Fully Connected

Floats
Out
32768
16384
16384
16384
16384
8192
8192
8192
8192
4096
4096
4096
10

4. THE EffNet MODEL

Additionally, applying max-pooling to the spatial convolution layer does not allow the network to properly encode the
data before it is reduced to a fourth of its incoming size. Nevertheless early stage pooling means cheaper following layers in
a block. In order to maintain the advantages of early pooling
while also relaxing data compression, we propose using separable pooling. Similar to separable convolution, we first apply
a 2 × 1 pooling kernel (with corresponding strides) after the
first spatial convolution layer. The second phase of the pooling
then follows the last pointwise convolution of the block.

4.1. Data Compression
Upon analysing the effects of the various methods discussed
in section 3, we established that small networks are very sensitive to data compression. Throughout the experiments, each
method which led to a larger bottleneck in data flow had also
resulted in decreased accuracies. For a better understanding
of the data flow concept Table 1 lists the dimensionality of an
input through the different stages of our Cifar 10 [23] network
comparison.

Residual Connections Initially proposed by [22] and
quickly adopted by many, residual connections have become
a standard practice. Papers like [22] showed that residual
connections are only beneficial in deeper networks. We extend
this claim and report a consistent decrease in accuracies when
using residual connections throughout our experiments. We
interpret this as a support to our claim that small networks
cannot handle large compression factors well.

4.2. The EffNet Blocks
We propose an efficient network structure which at the same
time solves the issue of data compression and implements the
insights from section 3. We design this block as a general
construction to seamlessly replace the vanilla convolutional
layers in, but not limited to, slim networks.
We start by, in a similar manner to [7], splitting the 3 × 3
depthwise convolution to two linear layers. This allows us to
pool after the first spatial layer, thus saving computations in
the second layer.
We then split the subsampling along the spatial dimensions.
As seen in Table 1 and in Figure 1 we apply a 1×2 max pooling
kernel after the first depthwise convolution. For the second
subsampling we choose to replace the common pointwise
convolution with 2 × 1 kernels and a corresponding stride.
This practically has the same amount of FLOPs yet leads to
slightly higher accuracies.
Following the preliminary experiments in section 3, we
decide to relax the bottleneck factor for the first pointwise
convolution. Instead of using a fourth of the output channels,
we recognise a factor of 0.5, with a minimal channel amount
of 6, as preferable.

Group Convolutions Following the promising results of [6]
we also experimented with similar configurations. The most
drastic structure here was the original ShuffleNet module and
the most relaxed configuration was the mere application of
group convolutions to the last point-wise layer in the blocks.
The results showed a clear decrease in accuracies. We have
therefore refrain from using group convolutions, despite the
computational benefits.
Addressing the First Layer Both MobileNet [5] and ShuffleNet [6] avoided replacing the first layer with the claim that
this layer is already rather cheap to begin with. We respectfully
disagree with this claim and believe that every optimisation
counts. After having optimised the rest of the layers in the
network, the first layer becomes proportionally larger. In our
experiments, replacing the first layer with our EffNet block
saves ∼ 30% of the computations for the respective layer.
3

5. EXPERIMENTS

ment which favour our EffNet model both in terms of accuracy
and FLOPs.

This section covers the evaluation of our model. We select
datasets which comply with our general settings; a small number of classes and a relatively small input resolution. We then
do a quick manual search for probable hyper-parameters for
the baseline which fulfil the requirements; two to three hidden
layers and small number of channels. The other architectures
then simply replace the convolutional layers without changing
the hyper-parameters.
Architectures were executed at least five times to cancel
out the effects of random initialisation.
To concentrate the experiments on our model, we did not
apply any sort of data augmentation or pre-training on additional data as proposed by [24]. Furthermore, we did not use
hyper-parameter optimisation as our goal is to replace the convolution layers in every given network with the EffNet blocks.
This makes our model simpler to use as it allows developers to
not worry about the implications of the swap to EffNet blocks.
We used Tensorflow [25] for all experiments. Networks
were trained using the Adam optimiser [26] with a learning
rate of 0.001 and β1 = 0.75.
As an additional experiment, we evaluate a larger EffNet
model with roughly the same amount of FLOPs as the baseline.
This is dubbed large in the following results.

Table 3: A comparison of the large and smaller on the SVHN dataset
Baseline
EffNet large

Mean Accuracy
91.08%
91.12%

kFLOPs
3,563.5
3,530.7

Factor
1.00
0.99

MobileNet
ShuffleNet
EffNet (ours)

85.64%
82.73%
88.51%

773.4
733.1
517.6

0.22
0.21
0.14

5.3. German Traffic Sign Recognition Benchmark
A slightly older dataset which is nevertheless very relevant
in most current driver assistance applications is the GTSRB
dataset [28]. With over 50, 000 images and some 43 classes it
presents a rather small task with a large variation in data and is
thus an interesting benchmark. As even small networks started
overfitting very quickly on this data, we resize the input images
to 32 × 32 and use a dropout [29] layer with drop-probability
of 50% before the output layer. Results are shown in Table 4
and also favour our EffNet model.
Table 4: A comparison of the large and smaller models on the
GTSRB dataset

Baseline
EffNet large

Mean Accuracy
94.48%
94.82%

kFLOPs
2,326.5
2,171.9

Factor
1.00
0.93

88.15%
88.99%
91.79%

533.0
540.7
344.1

0.23
0.23
0.15

Table 2: A comparison of the large and smaller on the Cifar10
dataset

Baseline
EffNet large

Mean Accuracy
82.78%
85.02%

Mil. FLOPs
80.3
79.8

Factor
1.00
0.99

MobileNet
ShuffleNet
EffNet (ours)

MobileNet
ShuffleNet
EffNet (ours)

77.48%
77.30%
80.20%

5.8
4.7
11.4

0.07
0.06
0.14

6. COMPARISON WITH MOBILENET V2
As [30] came out at the same time as our work, we extend this
work and write a quick comparison. Finally, we show how
using a few minor adjustments, we surpass [30] in terms of
accuracy while being similarly expensive to compute.

5.1. Cifar 10
Being a fundamental simple datasets in computer vision, Cifar10 [23] is a good example of the sort of tasks we aim to
improve on. Its images are small and represent a limited number of classes. We achieve a significant improvement over
MobileNet, ShuffleNet and even the baseline while still requiring ∼ 7 times less FLOPs than the baseline (see Table 2).
We relate this improvement to the additional depth of the network meaning that the EffNet blocks simulate a larger, deeper
network which does not underfit as much as the other models.

6.1. Architecture Comparison
Both [30] and this work separate the convolution operation
along some of its dimensions to save computations. Unlike
[30], we also separate the spatial, two dimensional kernels
into two single-dimensional kernels. We did notice a small
decrease in accuracies of around 0.5% across our experiments
by doing so, yet it allows for a significantly more efficient
implementation and requires less computations.
For tackling the data compression problem, [30] proposes
to significantly inflate the amount of data throughout their
block by multiplying the number of channels of the input by a
factor of 4 – 10. This makes the compression less aggressive
comparing to the respective block’s input while also moving it

5.2. Street View House Numbers
Similar to Cifar10, the SVHN benchmark [27] is also a common dataset for evaluation of simple networks. The data consists of 32 × 32 pixel patches centred around a digit with a
corresponding label. Table 3 shows the results of this experi4

to the end of the blocks, i.e. a reversed bottleneck. They further
recognise an interesting, often overlooked property of the
ReLU function. When following a Batch Normalisation layer,
ReLU sets half of the data to zero thus further compressing
the data. To counter the problem, [30] resolves to a linear
pointwise convolution at the end of each block. In practice
they get a linear layer followed by another non-linear pointwise
layer, i.e. B ∗ (A ∗ x) with x being the input, A the first layer
and B the second. Removing layer A altogether simply forces
the network to learn the layer B as the function B ∗ A. Our
experiments also showed an indifference to the existence of
layer A. Nevertheless, we show that using a leaky ReLU [31]
on top of the layer A significantly increases the performance.

performs favourably to [30] in most settings in terms of accuracy while having only marginally more FLOPs. Furthermore,
although the mob_imp model outperforms our model, it is
significantly more expensive to compute.
Table 5: A comparison of MobileNet v2 and EffNet on the Cifar10
dataset for various expansion rates
Ex. Rate
Baseline
EffNet
MobileNet v2
EffNet
MobileNet v2
EffNet
MobileNet v2
mob_imp

6
4
2

6.2. EffNet Adaptations

6

Considering latest experiments, we revise our architecture by
introducing three minor adjustments.
First, considering the bottleneck structure, we define the
output channels of the first pointwise layer as a function of the
block’s input channels rather than its output channels. Similar
to [30], yet less extreme, the number of channels is given by
b

Mean Acc.
82.78%
83.20%
79.10%
82.45%
78.91%
81.67%
76.47%
84.25%

Mil. Flops
80.3
44.1
42.0
31.1
29.2
18.1
16.4
44.0

Fact.
1.00
0.55
0.52
0.39
0.36
0.22
0.20
0.55

Table 6: A comparison of MobileNet v2 and EffNet on the SVHN
dataset for various expansion rates
Ex. Rate
6

inputChannels ∗ expansionRate
c
2

4
2

Second, the depth multiplier in the spatial convolution,
which we only previously increased in some cases, is now
natively integrated into our architecture and set to 2.
Last, we replace the ReLU on the pointwise layers with a
leaky ReLU.
Please note that the experiments were not decisive regarding both the activation function for the depthwise convolution
and the first layer in the network. For the sake of simplicity, we
used a ReLU with the remark that both leaky ReLU and linear
spatial convolution were occasionally preferable. The first
layer in the following experiments is a vanilla convolutional
layer with max pooling.

6

Baseline
EffNet
MobileNet v2
EffNet
MobileNet v2
EffNet
MobileNet v2
mob_imp

Mean Acc.
91.08%
87.80%
87.16%
87.49%
86.93%
87.30%
86.71%
88.78%

kFlops
3,563.5
2,254.8
2,130.4
1,729.5
1,646.6
1,204.2
1,162.8
2,506.7

Fact.
1.00
0.63
0.60
0.49
0.46
0.34
0.33
0.70

Table 7: A comparison of MobileNet v2 and EffNet on the GTSRB
dataset for various expansion rates
Ex. Rate
6
4
2

6.3. Experiments

6

We use the same datasets as in section 5 but aim at a different
kind of comparison. We now evaluate three models.

Baseline
EffNet
MobileNet v2
EffNet
MobileNet v2
EffNet
MobileNet v2
mob_imp

Mean Acc.
94.48%
93.74%
92.82%
92.30%
91.56%
90.40%
90.74%
93.25%

kFlops
2,326,5
1,208.3
1,159.2
956.4
934.9
704.5
710.7
1,408.0

Fact.
1.00
0.51
0.50
0.41
0.40
0.30
0.31
0.61

7. CONCLUSIONS
We have presented a novel convolutional block structure for
CNNs, called EffNet, which promises to significantly reduce
computational effort while preserving and even surpassing the
baseline’s accuracy. Our unified blocks are designed for safely
replacing the vanilla convolution layers in applications for
embedded and mobile hardware. As networks are reduced to a
small fraction of the baseline’s FLOPs, our method presents a
two-fold advantage, first is the quicker architecture and second
the application of a larger, deeper network becomes possible.
We have also showed how such a larger network is clearly
preferable to the baseline while requiring a similar number of
operations.

1. Our revised EffNet model
2. The original MobileNet v2 model
3. Our proposed modifications to the MobileNet v2 model
with pooling instead of strided convolution and leaky
ReLU instead of the linear bottleneck. This is dubbed
mob_imp (mobile improved) throughout the following
tables.
The models are evaluated with three different expansion
rates: 2, 4 and 6 while mob_imp is only tested with expansion
rate of 6. Tables 5, 6 and 7 show how our revised architecture
5

8. REFERENCES
[1] M. Babaeizadeh et al., “Noiseout: A simple way to prune
neural networks,” 2016.

[16] L. Lai, N. Suda, and V. Chandra, “Deep convolutional neural network inference with floating-point
weights and fixed-point activations,” arXiv preprint
arXiv:1703.03073, 2017.

[2] X. Dong, S. Chen, and S. J. Pan, “Learning to prune deep
neural networks via layer-wise optimal brain surgeon,”
2017.

[17] F. Chollet, “Xception: Deep learning with depthwise separable convolutions,” arXiv preprint arXiv:1610.02357,
2016.

[3] Pavlo Molchanov et al., “Pruning convolutional neural
networks for resource efficient transfer learning,” 2017.

[18] Max Jaderberg et al., “Speeding up convolutional neural
networks with low rank expansions,” in Proc. BMVC.
2014, BMVA Press.

[4] M. Rastegari et al., “Xnor-net: Imagenet classification
using binary convolutional neural networks,” in ECCV.
Springer, 2016, pp. 525–542.

[19] A. Krizhevsky et al., “Imagenet classification with deep
convolutional neural networks,” in Advances in neural
information processing systems, 2012, pp. 1097–1105.

[5] A. G. Howard et al., “Mobilenets: Efficient convolutional
neural networks for mobile vision applications,” arXiv
preprint arXiv:1704.04861, 2017.

[20] C. Tomasi and T. Kanade, Detection and tracking of
point features, School of Computer Science, Carnegie
Mellon Univ. Pittsburgh, 1991.

[6] X. Zhang et al., “Shufflenet: An extremely efficient
convolutional neural network for mobile devices,” arXiv
preprint arXiv:1707.01083, 2017.

[21] F. N. Iandola et al., “Squeezenet: Alexnet-level accuracy
with 50x fewer parameters and< 0.5 mb model size,”
arXiv preprint arXiv:1602.07360, 2016.

[7] C. Szegedy et al., “Rethinking the inception architecture
for computer vision,” in Proc. CVPR. IEEE, 2016, pp.
2818–2826.

[22] K. He et al., “Deep residual learning for image recognition,” in Proc. CVPR. IEEE, 2016, pp. 770–778.
[23] A. Krizhevsky and G. Hinton, “Learning multiple layers
of features from tiny images,” 2009.

[8] J. Snoek, H. Larochelle, and R. P. Adams, “Practical
bayesian optimization of machine learning algorithms,”
in Advances in neural information processing systems,
2012, pp. 2951–2959.

[24] J. Krause et al., “The unreasonable effectiveness of noisy
data for fine-grained recognition,” in ECCV. Springer,
2016, pp. 301–320.

[9] D. Horn and B. Bischl, “Multi-objective parameter configuration of machine learning algorithms using modelbased optimization,” in Symposium Series on Computational Intelligence. IEEE, 2016, pp. 1–8.

[25] M. Abadi et al., “Tensorflow: Large-scale machine
learning on heterogeneous distributed systems,” arXiv
preprint arXiv:1603.04467, 2016.
[26] D. Kingma and J. Ba, “Adam: A method for stochastic
optimization,” in Proc. ICLR. IEEE, 2015.

[10] J. Bergstra and Y. Bengio, “Random search for hyperparameter optimization,” Journal of Machine Learning
Research, vol. 13, pp. 281–305, 2012.

[27] Y. Netzer et al., “Reading digits in natural images with
unsupervised feature learning,” in NIPS, 2011, number 2,
p. 5.

[11] B. Zoph and Q. V. Le, “Neural architecture search with
reinforcement learning,” in ICLR. IEEE, 2017.

[28] Johannes S. et al., “The German Traffic Sign Recognition
Benchmark: A multi-class classification competition,” in
IJCNN. IEEE, 2011, pp. 1453–1460.

[12] S. Hochreiter and J. Schmidhuber, “Long short-term
memory,” Neural computation, vol. 9, no. 8, pp. 1735–
1780, 1997.
[13] E. Real et al., “Large-scale evolution of image classifiers,”
arXiv preprint arXiv:1703.01041, 2017.

[29] N. Srivastava et al., “Dropout: a simple way to prevent
neural networks from overfitting.,” Journal of machine
learning research, vol. 15, no. 1, pp. 1929–1958, 2014.

[14] X. Chen et al., “Fxpnet: Training a deep convolutional
neural network in fixed-point representation,” in IJCNN.
IEEE, 2017, pp. 2494–2501.

[30] Mark Sandler et al., “Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and
segmentation,” arXiv preprint arXiv:1801.04381, 2018.

[15] D. Lin, S. Talathi, and S. Annapureddy, “Fixed point
quantization of deep convolutional networks,” in ICML,
2016, pp. 2849–2858.

[31] Bing Xu et al., “Empirical evaluation of rectified activations in convolutional network,” ICML, 2015.

6

