Nitish Shirish Keskar
5834589
On Large Batch Training for Deep Learning: Generalization Gap and Sharp Minima
The stochastic gradient descent (SGD) method and its variants are algorithms of choice for many Deep Learning tasks. These methods operate in a small batch regime wherein a fraction of the training data, say $32$ $512$ data points, is sampled to compute an approximation to the gradient. It has been observed in practice that when using a larger batch there is a degradation in the quality of the model, as measured by its ability to generalize. We investigate the cause for this generalization drop in the large batch regime and present numerical evidence that supports the view that large batch methods tend to converge to sharp minimizers of the training and testing functions and as is well known, sharp minima lead to poorer generalization. In contrast, small batch methods consistently converge to flat minimizers, and our experiments support a commonly held view that this is due to the inherent noise in the gradient estimation. We discuss several strategies to attempt to help large batch methods eliminate this generalization gap.
ICLR
Nitish Shirish Keskar Dheevatsa Mudigere Jorge Nocedal Mikhail Smelyanskiy Ping Tak Peter Tang
212756
Regularizing and Optimizing LSTM Language Models
Recurrent neural networks (RNNs) such as long short term memory networks (LSTMs) serve as a fundamental building block for many sequence learning tasks, including machine translation, language modeling, and question answering. In this paper, we consider the specific problem of word level language modeling and investigate strategies for regularizing and optimizing LSTM based models. We propose the weight dropped LSTM which uses DropConnect on hidden to hidden weights as a form of recurrent regularization. Further, we introduce NT ASGD, a variant of the averaged stochastic gradient method, wherein the averaging trigger is determined using a non monotonic condition as opposed to being tuned by the user. Using these and other regularization strategies, we achieve state of the art word level perplexities on two data sets: 57.3 on Penn Treebank and 65.8 on WikiText 2. In exploring the effectiveness of a neural cache in conjunction with our proposed model, we achieve an even lower state of the art perplexity of 52.8 on Penn Treebank and 52.0 on WikiText 2.
ICLR
Stephen Merity Nitish Shirish Keskar Richard Socher
202573071
CTRL: A Conditional Transformer Language Model for Controllable Generation
Large scale language models show promising text generation capabilities, but users cannot easily control particular aspects of the generated text. We release CTRL, a 1.63 billion parameter conditional transformer language model, trained to condition on control codes that govern style, content, and task specific behavior. Control codes were derived from structure that naturally co occurs with raw text, preserving the advantages of unsupervised learning while providing more explicit control over text generation. These codes also allow CTRL to predict which parts of the training data are most likely given a sequence. This provides a potential method for analyzing large amounts of data via model based source attribution. We have released multiple full sized, pretrained versions of CTRL at this https URL.
ArXiv
Nitish Shirish Keskar Bryan McCann Lav R Varshney Caiming Xiong Richard Socher
49393754
The Natural Language Decathlon: Multitask Learning as Question Answering
Presented on August 28, 2018 at 12:15 p.m. in the Pettit Microelectronics Research Center, Room 102 A/B.
ArXiv
Bryan McCann Nitish Shirish Keskar Caiming Xiong Richard Socher
4111774
An Analysis of Neural Language Modeling at Multiple Scales
Many of the leading approaches in language modeling introduce novel, complex and specialized architectures. We take existing state of the art word level language models based on LSTMs and QRNNs and extend them to both larger vocabularies as well as character level granularity. When properly tuned, LSTMs and QRNNs achieve state of the art results on character level (Penn Treebank, enwik8) and word level (WikiText 103) datasets, respectively. Results are obtained in only 12 hours (WikiText 103) to 2 days (enwik8) using a single modern GPU.
ArXiv
Stephen Merity Nitish Shirish Keskar Richard Socher
201666437
Neural Text Summarization: A Critical Evaluation
Text summarization aims at compressing long documents into a shorter form that conveys the most important parts of the original document. Despite increased interest in the community and notable research effort, progress on benchmark datasets has stagnated. We critically evaluate key ingredients of the current research setup: datasets, evaluation metrics, and models, and highlight three primary shortcomings: 1) automatically collected datasets leave the task underconstrained and may contain noise detrimental to training and evaluation, 2) current evaluation protocol is weakly correlated with human judgment and does not account for important characteristics such as factual correctness, 3) models overfit to layout biases of current datasets and offer limited diversity in their outputs.
EMNLP/IJCNLP
Wojciech Kryscinski Nitish Shirish Keskar Bryan McCann Caiming Xiong Richard Socher
36899834
Improving Generalization Performance by Switching from Adam to SGD
Despite superior training outcomes, adaptive optimization methods such as Adam, Adagrad or RMSprop have been found to generalize poorly compared to Stochastic gradient descent (SGD) These methods tend to perform well in the initial portion of training but are outperformed by SGD at later stages of training. We investigate a hybrid strategy that begins training with an adaptive method and switches to SGD when appropriate. Concretely, we propose SWATS, a simple strategy which switches from Adam to SGD when a triggering condition is satisfied. The condition we propose relates to the projection of Adam steps on the gradient subspace. By design, the monitoring process for this condition adds very little overhead and does not increase the number of hyperparameters in the optimizer. We report experiments on several standard benchmarks such as: ResNet, SENet, DenseNet and PyramidNet for the CIFAR 10 and CIFAR 100 data sets, ResNet on the tiny ImageNet data set and language modeling with recurrent networks on the PTB and WT2 data sets. The results show that our strategy is capable of closing the generalization gap between SGD and Adam on a majority of the tasks.
ArXiv
Nitish Shirish Keskar Richard Socher
53107519
A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation
The convergence rate and final performance of common deep learning models have significantly benefited from heuristics such as learning rate schedules, knowledge distillation, skip connections, and normalization layers. In the absence of theoretical underpinnings, controlled experiments aimed at explaining these strategies can aid our understanding of deep learning landscapes and the training dynamics. Existing approaches for empirical analysis rely on tools of linear interpolation and visualizations with dimensionality reduction, each with their limitations. Instead, we revisit such analysis of heuristics through the lens of recently proposed methods for loss surface and representation analysis, viz. mode connectivity and canonical correlation analysis (CCA) and hypothesize reasons for the success of the heuristics. In particular, we explore knowledge distillation and learning rate heuristics of (cosine) restarts and warmup using mode connectivity and CCA. Our empirical analysis suggests that: (a) the reasons often quoted for the success of cosine annealing are not evidenced in practice; (b) that the effect of learning rate warmup is to prevent the deeper layers from creating training instability; and (c) that the latent knowledge shared by the teacher is primarily disbursed to the deeper layers.
ICLR
Akhilesh Deepak Gotmare Nitish Shirish Keskar Caiming Xiong Richard Socher
57375723
Coarse grain Fine grain Coattention Network for Multi evidence Question Answering
End to end neural models have made significant progress in question answering, however recent studies show that these models implicitly assume that the answer and evidence appear close together in a single document. In this work, we propose the Coarse grain Fine grain Coattention Network (CFC) a new question answering model that combines information from evidence across multiple documents. The CFC consists of a coarse grain module that interprets documents with respect to the query then finds a relevant answer, and a fine grain module which scores each candidate answer by comparing its occurrences across all of the documents with the query. We design these modules using hierarchies of coattention and self attention, which learn to emphasize different parts of the input. On the Qangaroo WikiHop multi evidence question answering task, the CFC obtains a new state of the art result of 70.6% on the blind test set, outperforming the previous best by 3% accuracy despite not using pretrained contextual encoders.
ICLR
Victor Zhong Caiming Xiong Nitish Shirish Keskar Richard Socher
53682389
Balancing Communication and Computation in Distributed Optimization
Methods for distributed optimization have received significant attention in recent years owing to their wide applicability in various domains including machine learning, robotics, and sensor networks. A distributed optimization method typically consists of two key components: communication and computation. More specifically, at every iteration (or every several iterations) of a distributed algorithm, each node in the network requires some form of information exchange with its neighboring nodes (communication) and the computation step related to a (sub) gradient (computation) The standard way of judging an algorithm via only the number of iterations overlooks the complexity associated with each iteration. Moreover, various applications deploying distributed methods may prefer a different composition of communication and computation. Motivated by this discrepancy, in this paper, we propose an adaptive cost framework that adjusts the cost measure depending on the features of various applications. We present a flexible algorithmic framework, where communication and computation steps are explicitly decomposed to enable algorithm customization for various applications. We apply this framework to the well known distributed gradient descent (DGD) method, and show that the resulting customized algorithms, which we call DGD<inline formula><tex math notation="LaTeX"^t$/tex math>/inline formula> NEAR DGD<inline formula><tex math notation="LaTeX"^t$/tex math>/inline formula> and NEAR DGD<inline formula><tex math notation="LaTeX"/tex math>/inline formula> compare favorably to their base algorithms, both theoretically and empirically. The proposed NEAR DGD<inline formula><tex math notation="LaTeX"/tex math>/inline formula> algorithm is an exact first order method where the communication and computation steps are nested, and when the number of communication steps is adaptively increased, the method converges to the optimal solution. We test the performance and illustrate the flexibility of the methods, as well as practical variants, on quadratic functions and classification problems that arise in machine learning, in terms of iterations, gradient evaluations, communications, and the proposed cost framework.
IEEE Transactions on Automatic Control
Albert S Berahas Raghu Bollapragada Nitish Shirish Keskar Ermin Wei