ner_tags: O O O O O O O O O O O O O B-Model I-Model O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B-Citation I-Citation I-Citation O B-Citation I-Citation I-Citation I-Citation O B-Citation I-Citation O O O O O O O O O O O O O O O B-Organization B-Software I-Software I-Software O O O O O O O O O O B-Model O O O O O O O O O O O B-Organization O O O O B-Dataset B-URL O O O O O O O O O O O O O O B-Citation I-Citation I-Citation I-Citation O O O O O O O B-Model O O O O O O O O O O O O O O O O O O O O O O O O B-Model O O O B-Model O O O O O O O O O O O O O O O O O O O B-Dataset I-Dataset I-Dataset O O O O O B-Model O O O O O B-Model O O O O O B-Citation I-Citation I-Citation I-Citation I-Citation I-Citation B-Model I-Model I-Model O O O O O O O B-Model O O O O O O O O O O O O O O O O O O O O B-Model B-Citation I-Citation I-Citation I-Citation O O O O O O O O O O O O O O O O O O O O O O B-Model O O O B-Model B-Citation I-Citation I-Citation I-Citation O B-Model O O O O O O O O O B-Model I-Model I-Model I-Model I-Model B-Model O O O O O O B-Model O O B-Model I-Model I-Model B-Model B-Citation I-Citation I-Citation O O O O O O O O O O O O O O O B-Model O O O O O O O O O O O O B-Dataset O O O O O O O O O O O O O O O O O O O O O O O O O O O B-Dataset I-Dataset I-Dataset O O O O O O O O O O O B-Model O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B-Model O O O O O O O O O O O O O
tokens: channels) with a 1 × 1 kernel. 3.3 Architectures The layers of a neural network make up its architecture. Designing a good architecture is difficult, time consuming, and requires expert knowledge. The hyperparameters, those that need to be defined before training, include the number of convolutional filters, filter size (typically 3 × 3 pixels), padding (which defines the margin used when the convolutions are applied), stride (which determines if any pixels are skipped during the convolution), pooling layer size and dropout (a random probability of a neuron being ignored) to name a few. Whilst much research has been done to choose the best hyperparameters (e.g. Simonyan Zisserman 2014 ; Szegedy et al. 2015 ; Murugan 2017 ), in practice it is trial and error, and complexity is added slowly. We use Google’s Tensorflow library 4 and three different architecture models (Table 1 ). Table 1. CNN architecture versions used in this paper. Note . a Taken from Google’s internal training on the ILSVRC-2012-CLS http://www.image-net.org/challenges/LSVRC/2012/ data set using single image crop and may differ from values listed elsewhere. Inception (Szegedy et al. 2015 ) is a state of the art CNN. The performance is often defined by the percentage of correct classifications featuring in the top-1 and top-5 predicted categories of each image. In 2015, Inception_v3 was the first CNN to bypass the average human top-5 error rate of 5 , obtaining a top-5 error of 3.5 on the ILSVRC-2012 data set of 1000 categories. The original Inception-v1 deep convolutional architecture was named GoogLeNet . It was inspired by Lin, Chen Yan ( 2013 )’s Network In Network , and was one of the first CNN architectures to implement modules of parallel operations (inception modules) instead of the traditional sequential stacking. We implement the latest version Inception_v4 (Szegedy et al. 2016 ) which has since been refined to include batch-normalization, additional factorization ideas, more inception modules, and a more simplified uniform architecture. Another CNN we use is NASNet-A-large (Zoph et al. 2017 ). Nasnet is an architecture that was created by a controller neural network Neural Architecture Search (NAS) . With a small data set, NAS uses a recurrent neural network (RNN; Zoph Le 2016 ) and reinforcement learning to continuously propose improvements to the architecture. Since the use of NAS on larger data sets would be computationally expensive, it was applied to CIFAR-10, a data set of 8 × 10 7 small images in 10 classes to search for an optimal but scalable convolutional cell. Using a larger data set Imagenet 2012 5 which consists of ∼1.4 × 10 7 images and 1000 classes, NASNet-A-large retrained the weights of its own repetitions of these cells and filters in the penultimate layer. The final architecture consists of 18 repeated cells with 168 convolutional filters per cell. NASNet-a-large exceeds the performance of any human-designed model to date. The last architecture we