knowledge representation
204838340
Contrastive Representation Distillation
Often we wish to transfer representational knowledge from one neural network to another. Examples include distilling a large network into a smaller one, transferring knowledge from one sensory modality to a second, or ensembling a collection of models into a single estimator. Knowledge distillation, the standard approach to these problems, minimizes the KL divergence between the probabilistic outputs of a teacher and student network. We demonstrate that this objective ignores important structural knowledge of the teacher network. This motivates an alternative objective by which we train a student to capture significantly more information in the teacher's representation of the data. We formulate this objective as contrastive learning. Experiments demonstrate that our resulting new objective outperforms knowledge distillation and other cutting edge distillers on a variety of knowledge transfer tasks, including single model compression, ensemble distillation, and cross modal transfer. Our method sets a new state of the art in many transfer tasks, and sometimes even outperforms the teacher network when combined with knowledge distillation. Code: this http URL
ICLR
Yonglong Tian Dilip Krishnan Phillip Isola
202583325
K BERT: Enabling Language Representation with Knowledge Graph
Pre trained language representation models, such as BERT, capture a general language representation from large scale corpora, but lack domain specific knowledge. When reading a domain text, experts make inferences with relevant knowledge. For machines to achieve this capability, we propose a knowledge enabled language representation model (K BERT) with knowledge graphs (KGs) in which triples are injected into the sentences as domain knowledge. However, too much knowledge incorporation may divert the sentence from its correct meaning, which is called knowledge noise (KN) issue. To overcome KN, K BERT introduces soft position and visible matrix to limit the impact of knowledge. K BERT can easily inject domain knowledge into the models by equipped with a KG without pre training by self because it is capable of loading model parameters from the pre trained BERT. Our investigation reveals promising results in twelve NLP tasks. Especially in domain specific tasks (including finance, law, and medicine) K BERT significantly outperforms BERT, which demonstrates that K BERT is an excellent choice for solving the knowledge driven problems that require experts.
AAAI
Weijie Liu Peng Zhou Zhe Zhao Zhiruo Wang Qi Ju Haotang Deng Ping Wang
54486075
Prior Knowledge Based Probabilistic Collaborative Representation for Visual Recognition
Collaborative representation is an effective way to design classifiers for many practical applications. In this paper, we propose a novel classifier, called the prior knowledge based probabilistic collaborative representation based classifier (PKPCRC) for visual recognition. Compared with existing classifiers which use the collaborative representation strategy, the proposed PKPCRC further includes characteristics of training samples of each class as prior knowledge. Four types of prior knowledge are developed from the perspectives of image distance and representation capacity. They adaptively accommodate the contribution of each class and result in an accurate representation to classify a query sample. Experiments and comparisons on four challenging databases demonstrate that PKPCRC outperforms several state of the art classifiers.
IEEE Transactions on Cybernetics
Rushi Lan Yicong Zhou Zhenbing Liu Xiaonan Luo
221894674
SentiLARE: Sentiment Aware Language Representation Learning with Linguistic Knowledge
Most of the existing pre trained language representation models neglect to consider the linguistic knowledge of texts, which can promote language understanding in NLP tasks. To benefit the downstream tasks in sentiment analysis, we propose a novel language representation model called SentiLARE, which introduces word level linguistic knowledge including part of speech tag and sentiment polarity (inferred from SentiWordNet) into pre trained models. We first propose a context aware sentiment attention mechanism to acquire the sentiment polarity of each word with its part of speech tag by querying SentiWordNet. Then, we devise a new pre training task called label aware masked language model to construct knowledge aware language representation. Experiments show that SentiLARE obtains new state of the art performance on a variety of sentiment analysis tasks.
EMNLP
Pei Ke Haozhe Ji Siyang Liu Xiaoyan Zhu Minlie Huang
207794278
SentiLARE: Linguistic Knowledge Enhanced Language Representation for Sentiment Analysis
Most of the existing pre trained language representation models neglect to consider the linguistic knowledge of texts, whereas we argue that such knowledge can promote language understanding in various NLP tasks. To benefit the downstream tasks in sentiment analysis, we propose a novel language representation model called SentiLR, which introduces word level linguistic knowledge including part of speech tag and prior sentiment polarity from SentiWordNet. During pre training, we first acquire the prior sentiment polarity of each word by querying the SentiWordNet dictionary with its part of speech tag. Then, we devise a new pre training task called label aware masked language model consisting of two sub tasks: 1) word knowledge recovering given the sentence level label; 2) sentence level label prediction with linguistic knowledge enhanced context. Experiments show that SentiLR achieves state of the art performances on sentence level aspect level sentiment analysis and sentiment aware data augmentation.
EMNLP
Pei Ke Haozhe Ji Siyang Liu Xiaoyan Zhu Minlie Huang
221662308
KRED: Knowledge Aware Document Representation for News Recommendations
News articles usually contain knowledge entities such as celebrities or organizations. Important entities in articles carry key messages and help to understand the content in a more direct way. An industrial news recommender system contains various key applications, such as personalized recommendation, item to item recommendation, news category classification, news popularity prediction and local news detection. We find that incorporating knowledge entities for better document understanding benefits these applications consistently. However, existing document understanding models either represent news articles without considering knowledge entities (e.g. BERT) or rely on a specific type of text encoding model (e.g. DKN) so that the generalization ability and efficiency is compromised. In this paper, we propose KRED, which is a fast and effective model to enhance arbitrary document representation with a knowledge graph. KRED first enriches entities' embeddings by attentively aggregating information from their neighborhood in the knowledge graph. Then a context embedding layer is applied to annotate the dynamic context of different entities such as frequency, category and position. Finally, an information distillation layer aggregates the entity embeddings under the guidance of the original document representation and transforms the document vector into a new one. We advocate to optimize the model with a multi task framework, so that different news recommendation applications can be united and useful information can be shared across different tasks. Experiments on a real world Microsoft News dataset demonstrate that KRED greatly benefits a variety of news recommendation applications.
RecSys
Danyang Liu Jianxun Lian Shiyin Wang Ying Qiao Jiun-Hung Chen Guang-zhong Sun Xing Xie
216642086
Exploiting Structured Knowledge in Text via Graph Guided Representation Learning
In this work, we aim at equipping pre trained language models with structured knowledge. We present two self supervised tasks learning over raw text with the guidance from knowledge graphs. Building upon entity level masked language models, our first contribution is an entity masking scheme that exploits relational knowledge underlying the text. This is fulfilled by using a linked knowledge graph to select informative entities and then masking their mentions. In addition we use knowledge graphs to obtain distractors for the masked entities, and propose a novel distractor suppressed ranking objective which is optimized jointly with masked language model. In contrast to existing paradigms, our approach uses knowledge graphs implicitly, only during pre training, to inject language models with structured knowledge via learning from raw text. It is more efficient than retrieval based methods that perform entity linking and integration during finetuning and inference, and generalizes more effectively than the methods that directly learn from concatenated graph triples. Experiments show that our proposed model achieves improved performance on five benchmark datasets, including question answering and knowledge base completion tasks.
EMNLP
Tao Shen Yi Mao Peng-cheng He Guodong Long Adam Trischler Weizhu Chen
203624739
Deep learning and deep knowledge representation in Spiking Neural Networks for Brain Computer Interfaces
OBJECTIVE This paper argues that Brain Inspired Spiking Neural Network (BI SNN) architectures can learn and reveal deep in time space functional and structural patterns from spatio temporal data. These patterns can be represented as deep knowledge, in a partial case in the form of deep spatio temporal rules. This is a promising direction for building new types of Brain Computer Interfaces called Brain Inspired Brain Computer Interfaces (BI BCI) A theoretical framework and its experimental validation on deep knowledge extraction and representation using SNN are presented. RESULTS The proposed methodology was applied in a case study to extract deep knowledge of the functional and structural organisation of the brain's neural network during the execution of a Grasp and Lift task. The BI BCI successfully extracted the neural trajectories that represent the dorsal and ventral visual information processing streams as well as its connection to the motor cortex in the brain. Deep spatiotemporal rules on functional and structural interaction of distinct brain areas were then used for event prediction in BI BCI. SIGNIFICANCE The computational framework can be used for unveiling the topological patterns of the brain and such knowledge can be effectively used to enhance the state of the art in BCI.
Neural Networks
Kaushalya Kumarasinghe Nikola K Kasabov Denise Taylor
208176103
Rule Guided Compositional Representation Learning on Knowledge Graphs
Representation learning on a knowledge graph (KG) is to embed entities and relations of a KG into low dimensional continuous vector spaces. Early KG embedding methods only pay attention to structured information encoded in triples, which would cause limited performance due to the structure sparseness of KGs. Some recent attempts consider paths information to expand the structure of KGs but lack explainability in the process of obtaining the path representations. In this paper, we propose a novel Rule and Path based Joint Embedding (RPJE) scheme, which takes full advantage of the explainability and accuracy of logic rules, the generalization of KG embedding as well as the supplementary semantic structure of paths. Specifically, logic rules of different lengths (the number of relations in rule body) in the form of Horn clauses are first mined from the KG and elaborately encoded for representation learning. Then, the rules of length 2 are applied to compose paths accurately while the rules of length 1 are explicitly employed to create semantic associations among relations and constrain relation embeddings. Moreover, the confidence level of each rule is also considered in optimization to guarantee the availability of applying the rule to representation learning. Extensive experimental results illustrate that RPJE outperforms other state of the art baselines on KG completion task, which also demonstrate the superiority of utilizing logic rules as well as paths for improving the accuracy and explainability of representation learning.
AAAI
Guanglin Niu Yongfei Zhang Bo Li Peng Cui Si Liu Jingyang Li Xiaowei Zhang
219051422
Evaluation of an interactive educational system in urban knowledge acquisition and representation based on students' profiles
The aim of this article is to study how students currently understand the conception of space through different media and how that understanding helps them to intervene in the space. First, the process of teaching and learning as well as innovative supporting technologies are analyzed, revealing the characteristics of the contemporary student profile and better ways of teaching according to it. Second, we describe the evaluation of an experiment with a virtual reality (VR) system for urban project design with students of architecture from two universities. The premise is that the technology used in VR is familiar to the current profile of students. This paper aims to study the advantages and disadvantages of this trend and to find a balance. To do so, we use a quantitative method to evaluate students' profiles and their level of satisfaction with the system. The results were obtained by a questionnaire and a survey, which show the role and use of technologies in the students' environment and the degree of satisfaction when using it in the educational processes. In line with our assumptions, the value of satisfaction in the use of an advanced visualization technology in the classroom reveals a high level of motivation in general with differentiation between students in their first and last phases of studies.
Expert Syst. J. Knowl. Eng.
Monica V Sanchez-Sepulveda David Fonseca Alicia Garcia Holgado Francisco Jose Garcia-Penalvo Jordi Franquesa Ernest Redondo Fernando Moreira