- Emma Strubell, Pat Verga, David Belanger, A. McCallum. 2017. Fast and Accurate Sequence Labeling with Iterated Dilated Convolutions. Abstract: Bi-directional LSTMs have emerged as a standard method for obtaining per-token vector representations serving as input to various token labeling tasks (whether followed by Viterbi prediction or independent classiﬁcation). This paper proposes an alternative to Bi-LSTMs for this purpose: iterated dilated convolutional neural networks (ID-CNNs), which have better capacity than traditional CNNs for large context and structured prediction. We describe a distinct combination of network structure, parameter sharing and training procedures that is not only more accurate than Bi-LSTM-CRFs, but also 8x faster at test time on long sequences. More-over, ID-CNNs with independent clas-siﬁcation enable a dramatic 14x test-time speedup, while still attaining accuracy comparable to the Bi-LSTM-CRF. We further demonstrate the ability of ID-CNNs to combine evidence over long sequences by demonstrating their improved accuracy on whole-document (rather than per-sentence) inference. Unlike LSTMs whose sequential processing on sentences of length N requires O ( N ) time even in the face of parallelism, IDCNNs permit ﬁxed-depth convolutions to run in parallel across entire documents. Today when many companies run basic NLP on the entire web and large-volume trafﬁc, faster methods are paramount to saving time and energy costs.
