tokens: function layers can be concatenated together to form a deep CNN. This is usually followed by a fully connected layer also known as a dense layer, which is then finally connected to the output layer. The output layer for a classification problem is the amount of class labels we have in our data. Additionally for deep CNNs, a dropout layer is typically added after the max-pooling operation, which randomly drops off a certain percentage of the preceding nodes. This is done so that the neural network learns to better generalize its performance across the entire data. This is used as a regularization technique to avoid overfitting. CNNs are also trained using back-propagation. However, in practice, since it is computationally difficult to calculate the gradient of the loss function for all images in the training data, we typically divide the data into mini-batches and use the stochastic gradient descent algorithm. Figure 2. Schematic diagram of a typical CNN. Here the input image is a grey scale image of size 48×48 and the output contains two nodes for the binary class labels. 8 and 16 filters of size 4×4 were used for the convolutional layers. These act as feature extractors that identify import patterns in the input image. This diagram was created using an open source tool called NN-SVG 5 (LeNail 2019 ). 2.4 Generative adversarial network (GAN) GANs are a class of machine learning algorithms (Goodfellow et al. 2014 ) in which two neural networks are trained simultaneously with opposing goals. They act against each other as adversaries in a minmax two-player game. A generative model G is tasked with generating new data that capture the distribution of the input data. A discriminator model D is tasked with classifying samples as either REAL (that belong to the original data distribution) or FAKE (samples generated by G ). Assuming is the value/loss function, we are interested in then the first term in equation ( 2 ), is the expectation of the logarithm of D ’s predictions when an input is from the real data sample. D ’s goal is to maximise this term. The second term represents one minus the expectation of the logarithm of D ’s predictions when data generated by G is passed on to to D . The goal of D is to maximize the second term but the goal of G is to minimize this term. This sets up the adversarial framework. In the ideal case, the generator perfectly samples the distribution of the input data distribution, and the discriminator output equals to 1/2. In practice however, we use a minibatch stochastic gradient descent method and train the generator and the discriminator alternatively. The algorithm and the proof for the convergence of the algorithm can be found in Goodfellow et al. ( 2014 ), however for the benefit of the reader, we briefly summarize the training of this network in algorithm 1 . Algorithm 1: Minibatch stochastic gradient descent algorithm used for training a GAN. GANs have been used successfully in a wide range of tasks ranging
ner_tags: O O O O O O O O O B-Model I-Model O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B-Model I-Model O O O O O O O O O O O O O O O O O O O O O O O O O O O B-Model I-Model O O O O O O O O O O O O O O O O O O O O B-Model O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B-Model O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B-Software I-Software B-Citation I-Citation O O B-Model I-Model I-Model B-Model B-Model O O O O O O O B-Citation I-Citation I-Citation I-Citation O O O O B-Model I-Model O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B-Citation I-Citation I-Citation I-Citation I-Citation I-Citation O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B-Model B-Model O O O O O O O O O O O