依存句法分析在深度学习中的应用

DCNNs

线性拼接

\label{eq:seq_con}
\widetilde{ \bf x}_{i,j} =    {\bf x}_i \oplus   {\bf x}_{i+1}\oplus \cdots \oplus  {\bf x}_{i+j}

DCNNs (Dependency-based CNN) (Ma et al. (2015))做了2种简单的改进，即基于路径和基于兄弟节点，如下图所示：

GCN

Graph Convolutional Networks

GCN是一种编码图数据的网络结构，给定一张$$n$$个节点的有向图，可以将其表示为邻接矩阵$$\bf A$$，其中$$A_{ij}=1$$表示存在从$$i$$$$j$$的边。在$$L$$层的GCN中，记第$$l$$层节点$$i$$的输入为$$h_i^{(l-1)}$$，输出为$$h_i^{(l)}$$。那么，图卷积操作定义如下： \begin{align} h_i^{(l)} = \sigma\big( \sum_{j=1}^n A_{ij} W^{(l)}{h}_j^{(l-1)} + b^{(l)} \big), \label{eqn:conv} \end{align}

$$A_{ij}$$只在邻接节点处等于$$1$$，所以图卷积实际上是让节点从邻接节点处获取总结性的信息。

soft pruning

其中，$$Q$$$$K$$ 都是上一层的表示 $$\mathbf{h}^{(l-1)}$$。用这些$$\mathbf{\tilde{A}}$$代替$$\mathbf{{A}}$$就可实现soft pruning了。这个机制的效果如下所示：

References

Rui Cai, Xiaodong Zhang, and Houfeng Wang. 2016. Bidirectional recurrent convolutional neural network for relation classification. In Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: Long papers), pages 756–765, Berlin, Germany, August. Association for Computational Linguistics.

Zhijiang Guo, Yan Zhang, and Wei Lu. 2019. Attention guided graph convolutional networks for relation extraction. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 241–251, Florence, Italy, July. Association for Computational Linguistics.

Yang Liu, Furu Wei, Sujian Li, Heng Ji, Ming Zhou, and Houfeng Wang. 2015. A dependency-based neural network for relation classification. In Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 2: Short papers), pages 285–290, Beijing, China, July. Association for Computational Linguistics.

Mingbo Ma, Liang Huang, Bowen Zhou, and Bing Xiang. 2015. Dependency-based convolutional neural networks for sentence embedding. In Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 2: Short papers), pages 174–179, Beijing, China, July. Association for Computational Linguistics.

Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 1: Long papers), pages 1556–1566, Beijing, China, July. Association for Computational Linguistics.

Yan Xu, Lili Mou, Ge Li, Yunchuan Chen, Hao Peng, and Zhi Jin. 2015. Classifying relations via long short term memory networks along shortest dependency paths. In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 1785–1794, Lisbon, Portugal, September. Association for Computational Linguistics.