How can information stored in knowledge bases be injected into language models task-specific model fine-tuning?
Injection knowledge during fine-tuning is a rela-
tively lightweight paradigm, where external knowl-
edge is often used to augment model inputs for
specific tasks (Zhou et al., 2019; Lin et al., 2019;
Liu et al., 2020b; Cheng et al., 2021; Kang et al.,
2022). When injecting unstructured textual knowl-
edge, some methods retrieve task-related informa-
tion from external corpora to augment the origi-
nal input text (Karpukhin et al., 2020; Liu et al.,
2020a). When using structured knowledge, such as
knowledge graphs, existing methods usually apply
knowledge representation learning methods (Bor-
des et al., 2013; Lin et al., 2015) to encode struc-
tured knowledge into embeddings, and then fuse
these knowledge embeddings with input token em-
beddings using knowledge injection methods (Sun
et al., 2020; Su et al., 2021; Yasunaga et al., 2021).
Fine-grained Fact Verification with Kernel Graph Attention Network
Fact Verification requires fine-grained natural language inference capability that finds subtle clues to identify the syntactical and semantically correct but not well-supported claims. This paper presents Kernel Graph Attention Network (KGAT), which conducts more fine-grained fact verification with kernel-based attentions. Given a claim and a set of potential evidence sentences that form an evidence graph, KGAT introduces node kernels, which better measure the importance of the evidence node, and edge kernels, which conduct fine-grained evidence propagation in the graph, into Graph Attention Networks for more accurate fact verification. KGAT achieves a 70.38% FEVER score and significantly outperforms existing fact verification models on FEVER, a large-scale benchmark for fact verification. Our analyses illustrate that, compared to dot-product attentions, the kernel-based attention concentrates more on relevant evidence sentences and meaningful clues in the evidence graph, which is the main source of KGAT’s effectiveness. All source codes of this work are available at https://github.com/thunlp/KernelGAT.
[{"first":"Zhenghao","last":"Liu","middle":[]},{"first":"Chenyan","last":"Xiong","middle":[]},{"first":"Maosong","last":"Sun","middle":[]},{"first":"Zhiyuan","last":"Liu","middle":[]}]
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
Fact verification (FV) is a challenging task which requires to retrieve relevant evidence from plain text and use the evidence to verify given claims. Many claims require to simultaneously integrate and reason over several pieces of evidence for verification. However, previous work employs simple models to extract information from evidence without letting evidence communicate with each other, e.g., merely concatenate the evidence for processing. Therefore, these methods are unable to grasp sufficient relational and logical information among the evidence. To alleviate this issue, we propose a graph-based evidence aggregating and reasoning (GEAR) framework which enables information to transfer on a fully-connected evidence graph and then utilizes different aggregators to collect multi-evidence information. We further employ BERT, an effective pre-trained language representation model, to improve the performance. Experimental results on a large-scale benchmark dataset FEVER have demonstrated that GEAR could leverage multi-evidence information for FV and thus achieves the promising result with a test FEVER score of 67.10%. Our code is available at https://github.com/thunlp/GEAR.
[{"first":"Jie","last":"Zhou","middle":[]},{"first":"Xu","last":"Han","middle":[]},{"first":"Cheng","last":"Yang","middle":[]},{"first":"Zhiyuan","last":"Liu","middle":[]},{"first":"Lifeng","last":"Wang","middle":[]},{"first":"Changcheng","last":"Li","middle":[]},{"first":"Maosong","last":"Sun","middle":[]}]
UnitedQA: A Hybrid Approach for Open Domain Question Answering
To date, most of recent work under the retrieval-reader framework for open-domain QA focuses on either extractive or generative reader exclusively. In this paper, we study a hybrid approach for leveraging the strengths of both models. We apply novel techniques to enhance both extractive and generative readers built upon recent pretrained neural language models, and find that proper training methods can provide large improvement over previous state-of-the-art models. We demonstrate that a simple hybrid approach by combining answers from both readers can efficiently take advantages of extractive and generative answer inference strategies and outperforms single models as well as homogeneous ensembles. Our approach outperforms previous state-of-the-art models by 3.3 and 2.7 points in exact match on NaturalQuestions and TriviaQA respectively.
[{"first":"Hao","last":"Cheng","middle":[]},{"first":"Yelong","last":"Shen","middle":[]},{"first":"Xiaodong","last":"Liu","middle":[]},{"first":"Pengcheng","last":"He","middle":[]},{"first":"Weizhu","last":"Chen","middle":[]},{"first":"Jianfeng","last":"Gao","middle":[]}]
KALA: Knowledge-Augmented Language Model Adaptation
Pre-trained language models (PLMs) have achieved remarkable success on various natural language understanding tasks. Simple fine-tuning of PLMs, on the other hand, might be suboptimal for domain-specific tasks because they cannot possibly cover knowledge from all domains. While adaptive pre-training of PLMs can help them obtain domain-specific knowledge, it requires a large training cost. Moreover, adaptive pre-training can harm the PLM’s performance on the downstream task by causing catastrophic forgetting of its general knowledge. To overcome such limitations of adaptive pre-training for PLM adaption, we propose a novel domain adaption framework for PLMs coined as Knowledge-Augmented Language model Adaptation (KALA), which modulates the intermediate hidden representations of PLMs with domain knowledge, consisting of entities and their relational facts. We validate the performance of our KALA on question answering and named entity recognition tasks on multiple datasets across various domains. The results show that, despite being computationally efficient, our KALA largely outperforms adaptive pre-training.
[{"first":"Minki","last":"Kang","middle":[]},{"first":"Jinheon","last":"Baek","middle":[]},{"first":"Sung Ju","last":"Hwang","middle":[]}]
Fine-grained Fact Verification with Kernel Graph Attention Network
Online contents with false information, such as fake news, political deception, and online rumors, have been growing significantly and spread widely over the past several years. How to automatically "fact check" the integrity of textual contents, to prevent the spread of fake news, and to avoid the undesired social influences of maliciously fabricated statements, is urgently needed for our society.
Fine-grained Fact Verification with Kernel Graph Attention Network
Recent research formulates this problem as the fact verification task, which targets to automatically verify the integrity of statements using trustworthy corpora, e.g., Wikipedia (Thorne et al., 2018a). For example, as shown in Figure 1, a system could first retrieve related evidence sentences from the background corpus, conduct joint reasoning over these sentences, and aggregate the signals to verify the claim integrity (Nie et al., 2019a;Yoneda et al., 2018;Hanselowski et al., 2018).
Fine-grained Fact Verification with Kernel Graph Attention Network
There are two challenges for evidence reasoning and aggregation in fact verification. One is that no ground truth evidence is given; the evidence sentences are retrieved from background corpora, which inevitably contain noise. The other is that the false claims are often deliberately fabricated; they may be semantically correct but are not supported. This makes fact verification a rather challenging task, as it requires the fine-grained reasoning ability to distinguish the subtle differences between truth and false statements . This paper presents a new neural structural reasoning model, Kernel Graph Attention Network (KGAT), that provides more fine-grained evidence selection and reasoning capability for fact verification using neural matching kernels (Xiong et al., 2017;Dai et al., 2018). Given retrieved evidence pieces, KGAT first constructs an evidence graph, using claim and evidence as graph nodes and fullyconnected edges. It then utilizes two sets of kernels, one on the edges, which selectively summarize clues for a more fine-grained node representation and propagate clues among neighbor nodes through a multi-layer graph attention; and the other on the nodes, which performs more accurate evidence selection by better matching evidence with the claim. These signals are combined by KGAT, to jointly learn and reason on the evidence graph for more accurate fact verification.
Fine-grained Fact Verification with Kernel Graph Attention Network
In our experiments on FEVER (Thorne et al., 2018a), a large-scale fact verification benchmark, KGAT achieves a 70.38% FEVER score, significantly outperforming previous BERT and Graph Neural Network (GNN) based approaches . Our experiments demonstrate KGAT's strong effectiveness especially on facts that require multiple evidence reasoning: our kernel-based attentions provide more sparse and focused attention patterns, which are the main source of KGAT's effectiveness.
Fine-grained Fact Verification with Kernel Graph Attention Network
This section describes our Kernel Graph Attention Network (KGAT) and its application in Fact Verification. Following previous research, KGAT first constructs an evidence graph using retrieved evidence sentences D = {e 1 , . . . , e p , . . . , e l } for claim c, and then uses the evidence graph to predict the claim label y (Sec. 3.1 and 3.2). As shown in Figure 2, the reasoning model includes two main components: Evidence Propagation with Edge Kernels (Sec. 3.3) and Evidence Selection with Node Kernels (Sec. 3.4).
Fine-grained Fact Verification with Kernel Graph Attention Network
Similar to previous research , KGAT constructs the evidence graph G by using each claim-evidence pair as a node and connects all node pairs with edges, making it a fullyconnected evidence graph with l nodes: N = {n 1 , . . . , n p , . . . , n l }.
Fine-grained Fact Verification with Kernel Graph Attention Network
KGAT unifies both multiple and single evidence reasoning scenarios and produces a probability P (y|c, D) to predict claim label y. Different from previous work , we follow the standard graph label prediction setting in graph neural network (Veličković et al., 2017) and split the prediction into two components: 1) the label prediction in each node conditioned on the whole graph P (y|n p , G); 2) the evidence selection probability P (n p |G): P (y|c, D) = l p=1 P (y|c, e p , D)P (e p |c, D),
Fine-grained Fact Verification with Kernel Graph Attention Network
(1) or in the graph notation:
Fine-grained Fact Verification with Kernel Graph Attention Network
The joint reasoning probability P (y|n p , G) calculates node label prediction with multiple evidence. The readout module (Knyazev et al., 2019) calculates the probability P (n p |G) and attentively combines per-node signals for prediction. The rest of this section describes the initialization of node representations (n p ) in Sec. 3.2, the calculation of per-node predictions P (y|n p , G) with Edge Kernels (Sec. 3.3), and the readout module P (n p |G) with Node Kernels (Sec. 3.4).
Fine-grained Fact Verification with Kernel Graph Attention Network
The node representations are initialized by feeding the concatenated sequence of claim, document (Wiki) title, and evidence sentence, to pre-trained BERT model (Devlin et al., 2019). Specifically, in the node n p , the claim and evidence correspond to m tokens (with "[SEP]") and n tokens (with Wikipedia title and "[SEP]") . Using the BERT encoder, we get the token hidden states H p with the given node n p :
Fine-grained Fact Verification with Kernel Graph Attention Network
The representation of the first token ("[CLS]") is denoted as the initial representation of node n p :
Fine-grained Fact Verification with Kernel Graph Attention Network
The rest of the sequences H p 1:m+n are also used to represent the claim and evidence tokens: H p 1:m for the claim tokens and H p m+1:m+n for the evidence tokens.
Fine-grained Fact Verification with Kernel Graph Attention Network
The evidence propagation and per-node label prediction in KGAT are conducted by Edge Kernels, which attentively propagate information among nodes in the graph G along the edges with the kernel attention mechanism.
Fine-grained Fact Verification with Kernel Graph Attention Network
Specifically, KGAT calculates the node n p 's representation v p with the kernel attention mechanism, and uses it to produce the per-node claim prediction y: v p = Edge-Kernel(n p , G), P (y|n p , G) = softmaxy(Linear(v p )).
Fine-grained Fact Verification with Kernel Graph Attention Network
The edge kernel of KGAT conducts a hierarchical attention mechanism to propagate information between nodes. It uses token level attentions to produce node representations and sentence level attentions to propagate information along edges.
Fine-grained Fact Verification with Kernel Graph Attention Network
Token Level Attention. The token level attention uses kernels to get the fine-grained representationẑ q→p of neighbor node n q , according to node n p . The content propagation and the attention are controlled by kernels.
Fine-grained Fact Verification with Kernel Graph Attention Network
To get the attention weight α q→p i for i-th token in n q , we first conduct a translation matrix M q→p between q-th node and p-th node. Each element of the translation matrix M q→p ij in M q→p is the cosine similarity of their corresponding tokens' BERT representations:
Fine-grained Fact Verification with Kernel Graph Attention Network
Then we use K kernels to extract the matching feature K(M q→p i ) from the translation matrix M q→p (Xiong et al., 2017;Dai et al., 2018;Qiao et al., 2019;MacAvaney et al., 2019):
Fine-grained Fact Verification with Kernel Graph Attention Network
Each kernel K k utilizes a Gaussian kernel to extract features and summarizes the translation score to support multi-level interactions:
Fine-grained Fact Verification with Kernel Graph Attention Network
where µ k and δ k are the mean and width for the k-th kernel, which captures a certain level of interactions between the tokens (Xiong et al., 2017). Then each token's attention weight α q→p i is calculated using a linear layer:
Fine-grained Fact Verification with Kernel Graph Attention Network
The attention weights are used to combine the token representations (ẑ q→p ):
Fine-grained Fact Verification with Kernel Graph Attention Network
which encodes the content signals to propagate from node n q to node n p . Sentence Level Attention. The sentence level attention combines neighbor node information to node representation v p . The aggregation is done by a graph attention mechanism, the same with previous work .
Fine-grained Fact Verification with Kernel Graph Attention Network
It first calculate the attention weight β q→p of n q node according to the p-th node n p :
Fine-grained Fact Verification with Kernel Graph Attention Network
where • denotes the concatenate operator and z p is the initial representation of n p . Then the p-th node's representation is updated by combining the neighbor node representationŝ z q→p with the attention:
Fine-grained Fact Verification with Kernel Graph Attention Network
It updates the node representation with its neighbors, and the updated information are selected first by the token level attention (Eq. 9) and then the sentence level attention (Eq. 11). Sentence Level Claim Label Prediction. The updated p-th node representation v p is used to calculate the claim label probability P (y|n p ):
Fine-grained Fact Verification with Kernel Graph Attention Network
The prediction of the label probability for each node is also conditioned on the entire graph G, as the node representation is updated by gather information from its graph neighbors.
Fine-grained Fact Verification with Kernel Graph Attention Network
The per-node predictions are combined by the "readout" function in graph neural networks , where KGAT uses node kernels to learn the importance of each evidence. It first uses node kernels to calculate the readout representation φ(n p ) for each node n p :
Fine-grained Fact Verification with Kernel Graph Attention Network
Similar to the edge kernels, we first conduct a translation matrix M c→e p between the p-th claim and evidence, using their hidden state set H p 1:m and H p m+1:m+n . The kernel match features K(M c→e p i ) on the translation matrix are combined to produce the node selection representation φ(n p ):
Fine-grained Fact Verification with Kernel Graph Attention Network
This representation is used in the readout to calculate p-th evidence selection probability P (n p |G): P (n p |G) = softmaxp(Linear(φ(n p ))).
Fine-grained Fact Verification with Kernel Graph Attention Network
KGAT leverages the kernels multi-level soft matching capability (Xiong et al., 2017) to weight the node-level predictions in the evidence graph based on their relevance with the claim:
Fine-grained Fact Verification with Kernel Graph Attention Network
The whole model is trained end-to-end by minimizing the cross entropy loss:
Fine-grained Fact Verification with Kernel Graph Attention Network
using the ground truth verification label y * .
Fine-grained Fact Verification with Kernel Graph Attention Network
This section describes the dataset, evaluation metrics, baselines, and implementation details in our experiments.  Table 1.
Fine-grained Fact Verification with Kernel Graph Attention Network
Evaluation Metrics. The official evaluation metrics 2 for claim verification include Label Accuracy (LA) and FEVER score. LA is a general evaluation metric, which calculates claim classification accuracy rate without considering retrieved evidence. The FEVER score considers whether one complete set of golden evidence is provided and better reflects the inference ability.
Fine-grained Fact Verification with Kernel Graph Attention Network
We also evaluate Golden FEVER (GFEVER) scores, which is the FEVER score but with golden evidence provided to the system, an easier setting. Precision, Recall and F1 are used to evaluate evidence sentence retrieval accuracy using the provided sentence level labels (whether the sentence is evidence or not to verify the claim).
Fine-grained Fact Verification with Kernel Graph Attention Network
Baselines. The baselines include top models during FEVER 1.0 task and BERT based models.
Fine-grained Fact Verification with Kernel Graph Attention Network
Three top models in FEVER 1.0 shared task are compared. Athene (Hanselowski et al., 2018) and UNC NLP (Nie et al., 2019a) utilize ESIM to encode claim evidence pairs. UCL MRG (Yoneda et al., 2018) leverages Convolutional Neural Network (CNN) to encode claim and evidence. These three models aggregate evidence by attention mechanism or label aggregation component.
Fine-grained Fact Verification with Kernel Graph Attention Network
The BERT based models are our main baselines, they significantly outperform previous methods without pre-training. BERT-pair, BERT-concat and GEAR are three baselines from the previous 2 https://github.com/sheffieldnlp/ fever-scorer  work . BERT-pair and BERTconcat regard claim-evidence pair individually or concatenate all evidence together to predict claim label. GEAR utilizes a graph attention network to extract supplement information from other evidence and aggregate all evidence through an attention layer. Soleimani et al. (2019); Nie et al. (2019b) are also compared in our experiments. They implement BERT sentence retrieval for a better performance. In addition, we replace kernel with dot product to implement our GAT version, which is similar to GEAR, to evaluate kernel's effectiveness. Implementation Details. The rest of this section describes our implementation details.
Fine-grained Fact Verification with Kernel Graph Attention Network
Document retrieval. The document retrieval step retrieves related Wikipedia pages and is kept the same with previous work (Hanselowski et al., 2018;Soleimani et al., 2019). For a given claim, it first utilizes the constituency parser in AllenNLP  to extract all phrases which potentially indicate entities. Then it uses these phrases as queries to find relevant Wikipedia pages through the online Me-diaWiki API 3 . Then the convinced article are reserved (Hanselowski et al., 2018).
Fine-grained Fact Verification with Kernel Graph Attention Network
Sentence retrieval. The sentence retrieval part focuses on selecting related sentences from retrieved pages. There are two sentence retrieval models in our experiments: ESIM based sentence retrieval and BERT based sentence retrieval. The ESIM based sentence retrieval keeps the same as the previous work (Hanselowski et al., 2018;. The base version of BERT is used to implement our BERT based sentence retrieval model. We use the "[CLS]" hidden state to represent claim and evidence sentence pair. Then a learning to rank layer is leveraged to project "[CLS]" hidden state to ranking score. Pairwise loss is used to optimize the ranking model. Some work (Zhao et al., 2020;Ye et al., 2020) also employs our BERT based sentence retrieval in their experiments.
Fine-grained Fact Verification with Kernel Graph Attention Network
Claim verification. During training, we set the  batch size to 4 and accumulate step to 8. All models are evaluated with LA on the development set and trained for two epochs. The training and development sets are built with golden evidence and higher ranked evidence with sentence retrieval. All claims are assigned with five pieces of evidence. The BERT (Base), BERT (Large) and RoBERTa  are evaluated in claim verification.
Fine-grained Fact Verification with Kernel Graph Attention Network
In our experiments, the max length is set to 130. All models are implemented with PyTorch. BERT inherits huggingface's implementation 4 . Adam optimizer is used with learning rate = 5e-5 and warm up proportion = 0.1. The kernel size is set to 21, the same as previous work (Qiao et al., 2019).
Fine-grained Fact Verification with Kernel Graph Attention Network
The experiments are conducted to study the performance of KGAT, its advantages on different reasoning scenarios, and the effectiveness of kernels.
Fine-grained Fact Verification with Kernel Graph Attention Network
The fact verification performances are shown in Table 2. Several testing scenarios are conducted to compare KGAT effectiveness to BERT based baselines: BERT (Base) Encoder with ESIM retrieved sentences, with BERT retrieved sentences, and BERT (Large) Encoder with BERT retrieved sentences.
Fine-grained Fact Verification with Kernel Graph Attention Network
Compared with baseline models, KGAT is the best on all testing scenarios. With ESIM sentence retrieval, same as the previous work Hanselowski et al., 2018), KGAT outperforms the graph attention models GEAR and our GAT on both development and testing sets.  It illustrates the effectiveness of KGAT among graph based reasoning models. With BERT based sentence retrieval, our KGAT also outperforms BERT (Base) (Soleimani et al., 2019) by almost 1% FEVER score, showing consistent effectiveness with different sentence retrieval models. When using BERT (Large) as the encoder, KGAT also outperforms the corresponding version of Soleimani et al. (2019). KGAT with RoBERTa performs the best compared with all previously published research on all evaluation metrics. CorefBERT (Ye et al., 2020) extends our KGAT architecture and explicitly models co-referring relationship in context for better performance. The sentence retrieval performances of ESIM and BERT are compared in Table 3. The BERT sentence retrieval outperforms ESIM sentence retrieval significantly, thus also helps improve KGAT's reasoning accuracy. Nevertheless, for more fair comparisons, our following experiments are all based on ESIM sentence retrieval, which is the one used by GEAR, our main baseline .
Fine-grained Fact Verification with Kernel Graph Attention Network
This experiment studies the effectiveness of kernel on multiple and single evidence reasoning scenarios, as well as the contribution of kernels.
Fine-grained Fact Verification with Kernel Graph Attention Network
The verifiable instances are separated (except instances with "NOT ENOUGH INFO" label ) into two groups according to the golden evidence labels. If more than one evidence pieces are required, the claim is considered as requiring multi-evidence reasoning. The single evidence reasoning set and the multiple evidence reasoning set contain 11,372 (85.3%) and 1,960 (14.7%) instances, respectively. We also evaluate two additional KGAT variations: KGAT-Node which only uses kernels on the node, with the edge kernels replaced by standard dotproduction attention, and KGAT-Edge which only uses kernels on the edge. The results of these systems on the two scenarios are shown in Table 4.
Fine-grained Fact Verification with Kernel Graph Attention Network
KGAT-Node outperforms GAT by more than 0.3% on both single and multiple reasoning sce-  narios. As expected, it does not help much on GFEVER, because the golden evidence is given and node selection is not required. It illustrates KGAT-Node mainly focuses on choosing appropriate evidence and assigning accurate combining weights in the readout.
Fine-grained Fact Verification with Kernel Graph Attention Network
KGAT-Edge outperforms GAT by more than 0.8% and 0.1% on multiple and single evidence reasoning scenarios, respectively. Its effectiveness is mostly on combining the information from multiple evidence pieces.
Fine-grained Fact Verification with Kernel Graph Attention Network
The multiple and single evidence reasoning scenarios evaluate the reasoning ability from different aspects. The single evidence reasoning mainly focuses on selecting the most relevant evidence and inference with single evidence. It mainly evaluates model de-noising ability with the retrieved evidence. The multiple evidence reasoning is a harder and more complex scenario, requiring models to summarize necessary clues and reason over multiple evidence. It emphasizes to evaluate the evidence interactions for the joint reasoning. KGAT-Node shows consistent improvement on both two reasoning scenarios, which demonstrates the important role of evidence selection. KGAT-Edge, on the other hand, is more effective on multiple reasoning scenarios as the Edge Kernels help better propagate information along the edges.
Fine-grained Fact Verification with Kernel Graph Attention Network
This set of experiments further illustrate the influences of kernels in KGAT.
Fine-grained Fact Verification with Kernel Graph Attention Network
More Concentrated Attention. This experiment studies kernel attentions by their entropy, which reflects whether the learned attention weights are focused or scattered. The entropy of the kernel attentions in KGAT, the dot-product at-(a) Edge Attention.
Fine-grained Fact Verification with Kernel Graph Attention Network
(b) Node Attention.  tentions in GAT, and the uniform attentions are shown in Figure 3. The entropy of Edge attention is shown in Figure 3(a). Both GAT and KGAT show a smaller entropy of the token attention than the uniform distribution. It illustrates that GAT and KGAT have the ability to assign more weight to some important tokens with both dot product based and kernel based attentions. Compared to the dot-product attentions in GAT, KGAT's Edge attention focuses on fewer tokens and has a smaller entropy.
Fine-grained Fact Verification with Kernel Graph Attention Network
The entropy of Node attentions are plotted in Figure 3(b). GAT's attentions distribute almost the same with the uniform distribution, while KGAT has concentrated Node attentions on a few evidence sentences. As shown in the next experiment, the kernel based node attentions focus on the correct evidence pieces and de-noises the retrieved sentences, which are useful for claim verification.
Fine-grained Fact Verification with Kernel Graph Attention Network
More Accurate Evidence Selection. This experiment evaluates the effectiveness of KGAT-Node through attention distribution and evidence recall. The results are shown in Figure 4.
Fine-grained Fact Verification with Kernel Graph Attention Network
We first obtain the node attention score in the evidence graph from KGAT or GAT, and calcu-  late the statistics of the maximum one for each claim, as most of which only require single evidence to verify. The attention score of the highest attended evidence node for each claim is plotted in Figure 4(a). As expected, KGAT concentrates its weight to select evidence nodes and provides a focused attention.
Fine-grained Fact Verification with Kernel Graph Attention Network
Then the evidence selection accuracy is evaluated by their evidence recall. We first rank all evidence pieces for each claim. Then the evidence recall with different ranking depths is plotted in Figure 4(b). KGAT achieves a much higher recall on top ranking positions-only the first ranked sentence covers nearly 80% of ground truth evidence, showing the node kernels' ability to select correct evidence. This also indicates the potential of the node kernels in the sentence retrieval stage, which we reserve for future work as this paper focuses on the reasoning stage.
Fine-grained Fact Verification with Kernel Graph Attention Network
Fine-Grained Evidence Propagation. The third analysis studies the distribution of KGAT-Edge's attention which is used to propagate the evidence clues in the evidence graph. Figure 5 plots the attention weight distribution of the edge attention scores in KGAT and GAT, one from kernels and one from dot-products. The kernel attentions again are more concentrated: KGAT focuses fewer words while GAT's dot-product attentions are almost equally distributed among all words. This observation of the scattered dotproduct attention is consistent with previous research (Clark et al., 2019). As shown in the next case study, the edge kernels provide a fine-grained and intuitive attention pattern when combining evidence clues from multiple pieces. (1) [Al Jardine] Alan Charles Jardine (born September 3, 1942) is an American musician, singer and songwriter who co-founded the Beach Boys.
Fine-grained Fact Verification with Kernel Graph Attention Network
(2) [Al Jardine] He is best known as the band's rhythm guitarist, and for occasionally singing lead vocals on singles such as "Help Me, Rhonda" (1965), "Then I Kissed Her" (1965) and "Come Go with Me" (1978).  6 Case Study Table 5 shows the example claim used in GEAR  and the evidence sentences retrieved by ESIM, among which the first two are required evidence pieces. Figure 6 presents the distribution of attentions from the first evidence to the tokens in the second evidence (α 2→1 i ) in KGAT (Edge Kernel) and GAT (dot-product).
Fine-grained Fact Verification with Kernel Graph Attention Network
The first evidence verifies that "Al Jardine is an American musician" but does not enough information about whether "Al Jardine is a rhythm guitarist". The edge kernels from KGAT accurately pick up the additional information evidence (1) required from evidence (2): "rhythm guitarist". It effectively fills the missing information and completes the reasoning chain. Interesting, "Al Jardine" also receives more attention, which helps to verify if the information in the second evidence is about the correct person. This kernel attention pattern is more intuitive and effective than the dot-product attention in GAT. The later one scatters almost uniformly across all tokens and hard to explain how the joint reasoning is conducted. This seems to be a common challenge of the dot-product attention in Transformers (Clark et al., 2019).
Fine-grained Fact Verification with Kernel Graph Attention Network
This paper presents KGAT, which uses kernels in Graph Neural Networks to conduct more accurate evidence selection and fine-grained joint reasoning. Our experiments show that kernels lead to the more accurate fact verification. Our studies illustrate the two kernels play different roles and contribute to different aspects crucial for fact verification. While the dot-product attentions are rather scattered and hard to explain, the kernel-based attentions show intuitive and effective attention patterns: the node kernels focus more on the correct evidence pieces; the edge kernels accurately gather the necessary information from one node to the other to complete the reasoning chain. In the future, we will further study this properties of kernel-based attentions in neural networks, both in the effectiveness front and also the explainability front.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
Due to the rapid development of information extraction (IE), huge volumes of data have been extracted.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
How to automatically verify the data becomes a vital problem for various datadriven applications, e.g., knowledge graph completion (Wang et al., 2017) and open domain question answering (Chen et al., 2017a). Hence, many recent research efforts have been devoted to fact verification (FV), which aims to verify given claims with the evidence retrieved from plain text. † Corresponding author: Z.Liu(liuzy@tsinghua.edu.cn) "SUPPORTED" Example Claim
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
The Rodney King riots took place in the most populous county in the USA.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
(1) The 1992 Los Angeles riots, also known as the Rodney King riots were a series of riots, lootings, arsons, and civil disturbances that occurred in Los Angeles County, California in April and May 1992.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
(2) Los Angeles County, officially the County of Los Angeles, is the most populous county in the USA.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
"REFUTED" Example Claim Giada at Home was only available on DVD.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
(1) Giada at Home is a television show and first aired on October 18, 2008, on the Food Network.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
(2) Food Network is an American basic cable and satellite television channel. More specifically, given a claim, an FV system is asked to label it as "SUPPORTED", "REFUTED", or "NOT ENOUGH INFO", which indicate that the evidence can support, refute, or is not sufficient for the claim.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
Existing FV methods formulate FV as a natural language inference (NLI) (Angeli and Manning, 2014) task. However, they utilize simple evidence combination methods such as concatenating the evidence or just dealing with each evidence-claim pair. These methods are unable to grasp sufficient relational and logical information among the evidence. In fact, many claims require to simultaneously integrate and reason over several pieces of evidence for verification. As shown in Table 1, for both of the "SUPPORTED" example and "REFUTED" example, we cannot verify the given claims via checking any evidence in isolation. The claims can be verified only by understanding and reasoning over the multiple evidence.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
To integrate and reason over information from multiple pieces of evidence, we propose a graph-based evidence aggregating and reasoning (GEAR) framework. Specifically, we first build a fully-connected evidence graph and encourage information propagation among the evidence. Then, we aggregate the pieces of evidence and adopt a classifier to decide whether the evidence can support, refute, or is not sufficient for the claim. Intuitively, by sufficiently exchanging and reasoning over evidence information on the evidence graph, the proposed model can make the best of the information for verifying claims. For example, by delivering the information "Los Angeles County is the most populous county in the USA" to "the Rodney King riots occurred in Los Angeles County" through the evidence graph, the synthetic information can support "The Rodney King riots took place in the most populous county in the USA". Furthermore, we adopt an effective pretrained language representation model BERT (Devlin et al., 2019) to better grasp both evidence and claim semantics.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
We conduct experiments on the large-scale benchmark dataset for Fact Extraction and VERification (FEVER) (Thorne et al., 2018a). Experimental results show that the proposed framework outperforms recent state-of-the-art baseline systems. The further case study indicates that our framework could better leverage multi-evidence information and reason over the evidence for FV.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
The FEVER shared task (Thorne et al., 2018b) challenges participants to develop automatic fact verification systems to check the veracity of human-generated claims by extracting evidence from Wikipedia. The shared task is hosted as a competition on Codalab 1 with a blind test set. Nie et al. (2019); Yoneda et al. (2018) and Hanselowski et al. (2018) have achieved the top three results among 23 teams.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
Existing methods mainly formulate FV as an NLI task. Thorne et al. (2018a) simply concatenate all evidence together, and then feed the concatenated evidence and the given claim into the NLI model. Luken et al. (2018) adopt the decomposable attention model (DAM) (Parikh et al., 2016) to generate NLI predictions for each claimevidence pair individually and then aggregate all NLI predictions for final verification. Then, Hanselowski et al. (2018); Yoneda et al. (2018); Hidey and Diab (2018) adopt the enhanced sequential inference model (ESIM) (Chen et al., 2017b), a more effective NLI model, to infer the relevance between evidence and claims instead of DAM. As pre-trained language models have achieved great results on various NLP applications, Malon (2018) fine-tunes the generative pretraining transformer (GPT) (Radford et al., 2018) for FV. Based on the methods mentioned above, Nie et al. (2019) specially design the neural semantic matching network (NSMN), which is a modification of ESIM and achieves the best results in the competition. Unlike these methods, Yin and Roth (2018) propose the TWOWINGOS system which trains the evidence identification and claim verification modules jointly.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
The natural language inference (NLI) task requires a system to label the relationship between a pair of premise and hypothesis as entailment, contradiction or neutral. Several large-scale datasets have been proposed to promote the research in this direction, such as SNLI (Bowman et al., 2015) and Multi-NLI (Williams et al., 2018). These datasets have made it feasible to train complicated neural models which have achieved the state-of-the-art results (Bowman et al., 2015;Parikh et al., 2016;Sha et al., 2016;Chen et al., 2017b,c;Munkhdalai and Yu, 2017;Nie and Bansal, 2017;Conneau et al., 2017;Gong et al., 2018;Tay et al., 2018;Ghaeini et al., 2018). It is intuitive to transfer NLI models into the claim verification stage of the FEVER task and several teams from the shared task have achieved promising results by this way.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
Pre-trained language representation models such as ELMo  and OpenAI GPT (Radford et al., 2018) are proven to be effective on many NLP tasks. BERT (Devlin et al., 2019) employs bidirectional transformer and welldesigned pre-training tasks to fuse bidirectional context information and obtains the state-of-theart results on the NLI task. In our experiments, we find the fine-tuned BERT model outperforms other NLI-based models on the claim verification subtask of FEVER. Hence, we use BERT as the sentence encoder in our framework to better encoding semantic information of evidence and claims. 
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
We employ a three-step pipeline with components for document retrieval, sentence selection and claim verification to solve the task. In the document retrieval and sentence selection stages, we simply follow the method from Hanselowski et al. (2018) since their method has the highest score on evidence recall in the former FEVER shared task. And we propose our Graph-based Evidence Aggregating and Reasoning (GEAR) framework in the final claim verification stage. The full pipeline of our method is illustrated in Figure 1.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
In this section, we describe our document retrieval and sentence selection components. Additionally, we add a threshold filter after the sentence selection component to filter out those noisy evidence.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
In the document retrieval step, we adopt the entity linking approach from Hanselowski et al. (2018). Given a claim, the method first utilizes the constituency parser from AllenNLP  to extract potential entities from the claim. Then it uses the entities as search queries and finds relevant Wikipedia documents via the online MediaWiki API 2 . The seven highest-ranked results for each query are stored to form a candidate article set. Finally, the method drops the articles which are not in the offline Wikipedia dump and filters the articles by the word overlap between their titles and the claim.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
The sentence selection component selects the most relevant evidence for the claim from all sentences in the retrieved documents. Hanselowski et al. (2018) modify the ESIM model to compute the relevance score between the evidence and the claim. In the training phase, the model uses the hinge loss function max(0, 1 + s n −s p ) with the negative sampling strategy, where s p and s n denote the relevance scores of positive and negative samples. In the test phase, the final model ensembles the results from 10 models with different random seeds. Sentences with top-5 relevance scores are selected to form the final evidence set in the original method.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
In addition to the original model (Hanselowski et al., 2018), we add a relevance score filter with a threshold τ . Sentences with relevance scores lower than τ are filtered out to alleviate the noises. Thus the final size of the retrieved evidence set is equal to or less than 5. We choose different values of τ and select the value based on the dev set result. The evaluation results of the document retrieval and sentence selection components are shown in Section 5.1.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
In this section, we describe our GEAR framework for claim verification. As shown in Figure 1, given a claim and the retrieved evidence, we first utilize a sentence encoder to obtain representations for the claim and the evidence. Then we build a fully-connected evidence graph and propose an evidence reasoning network (ERNet) to propagate information among evidence and reason over the graph. Finally, we utilize an evidence aggregator to infer the final results.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
Given an input sentence, we employ BERT (Devlin  Specifically, given a claim c and N pieces of retrieved evidence {e 1 , e 2 , ..., e N }, we feed each evidence-claim pair (e i , c) into BERT to obtain the evidence representation e i . We also feed the claim into BERT alone to obtain the claim presentation c. That is,
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
(1)
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
Note that we concatenate the evidence and the claim to extract the evidence representation because the evidence nodes in the reasoning graph need the information from the claim to guide the message passing process among them.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
To encourage the information propagation among evidence, we build a fully-connected evidence graph where each node indicates a piece of evidence. We also add self-loop to every node because each node needs the information from itself in the message propagation process. We use
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
to represent the hidden states of nodes at layer t, where h t i ∈ R F ×1 and F is the number of features in each node. The initial hidden state of each evidence node h 0 i is initialized by the evidence presentation: h 0 i = e i . Inspired by recent work on semi-supervised graph learning and relational reasoning (Kipf and Welling, 2017;Velickovic et al., 2018;Palm et al., 2018), we propose an evidence reasoning network (ERNet) to propagate information among the evidence nodes. We first use an MLP to compute the attention coefficients between a node i and its neighbor j (j ∈ N i ),
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
where N i denotes the set of neighbors of node i, W t−1 0 ∈ R H×2F and W t−1 1 ∈ R 1×H are weight matrices, and · · denotes concatenation operation.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
Then, we normalize the coefficients using the softmax function,
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
Finally, the normalized attention coefficients are used to compute a linear combination of the neighbor features and thus we obtain the features for node i at layer t,
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
By stacking T layers of ERNet, we assume that each evidence could grasp enough information by communicating with other evidence. We feed the final hidden states of evidence nodes {h T 1 , h T 2 , ..., h T N } into our evidence aggregator to make the final inference.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
We employ an evidence aggregator to gather information from different evidence nodes and obtain the final hidden state o ∈ R F ×1 . The aggregator may utilize different aggregating strategies and we suggest three aggregators in our framework:
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
Attention Aggregator. Here we use the representation of the claim c to attend the hidden states of evidence and get the final aggregated state o.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
where W 0 ∈ R H×2F and W 1 ∈ R 1×H . Max Aggregator. The max aggregator performs the element-wise Max operation among hidden states.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
Mean Aggregator. The mean aggregator performs the element-wise Mean operation among hidden states.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
Once the final state o is obtained, we employ a one-layer MLP to get the final prediction l.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
where W ∈ R C×F and b ∈ R C×1 are parameters, and C is the number of prediction labels.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
We conduct our experiments on the large-scale dataset FEVER (Thorne et al., 2018a). The dataset consists of 185,455 annotated claims with a set of 5,416,537 Wikipedia documents from the June 2017 Wikipedia dump. We follow the dataset partition from the FEVER Shared Task (Thorne et al., 2018b). Table 2 shows the statistics of the dataset.  
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
In this section, we describe the baseline systems in our experiments. We first introduce the top-3 systems from the FEVER shared task. As BERT (Devlin et al., 2019) has achieved promising performance on several NLP tasks, we also implement two baseline systems via fine-tuning BERT in the claim verification task.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
We choose the top-3 models from the FEVER shared task as our baselines. The Athene UKP TU Darmstadt team (Athene) (Hanselowski et al., 2018) combines five inference vectors from the ESIM model via attention mechanism to make the final prediction.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
The UCL Machine Reading Group (UCL MRG) (Yoneda et al., 2018) predicts the label of each evidence-claim pair and aggregates the results via a label aggregation component.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
The UNC NLP team (Nie et al., 2019) proposes the neural semantic matching network and uses the model jointly to solve all three subtasks. They also incorporate additional information such as pageview frequency and WordNet features. They have achieved best results in the competition.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
We implement two BERT fine-tuning systems with different evidence combination approaches. The BERT-Concat system concatenates all evidence into a single string while the BERT-Pair system encodes each evidence-claim pair independently and then aggregates the results. Both systems share the same document retrieval and sentence selection components proposed by us.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
BERT-Concat. In the BERT-Concat system, we simply concatenate all evidence into a single sentence and utilize BERT to predict the relation between the concatenated evidence and the claim. In the training phase, we add the ground truth evidence into the retrieved evidence set with relevance score 1 and select five pieces of evidence with the highest scores. In the test phase, we concatenate the retrieved evidence for predicting.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
BERT-Pair. In the BERT-Pair system, we utilize BERT to predict the label for each evidenceclaim pair. Concretely, we use each evidenceclaim pair as the input and the label of the claim as the prediction target. In the training phase, we select the ground truth evidence for SUPPORTED and REFUTED claims and the retrieved evidence for NEI claims. In the test phase, we predict labels for all retrieved evidence-claim pairs. Because different evidence-claim pairs may have inconsistent predicted labels, we then utilize an aggregator to obtain the final claim label. We find the aggregator only returning the predicted label from the most relevant evidence has the best performance.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
We utilize BERT BASE (Devlin et al., 2019) in all of the BERT fine-tuning baselines and our GEAR framework. The learning rate is 2e-5.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
For BERT-Concat, the maximum sequence length is 256 and the batch size is 16. We limit the max length for concatenated evidence to 240 and the max length for claims to 16. We train this model for two epochs based on dev results. For BERT-Pair, we set the maximum sequence length to 128 and batch size to 32. We train this model for one epoch. As for the GEAR framework, we use the fine-tuned BERT-Pair model to extract features and the batch size is 512.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
In our ERNet, we set the batch size to 256, the number of features F to 768 and the dimension of weight matrices H to 64. The model is trained to minimize the negative log likelihood loss on the predicted label using the Adam optimizer (Kingma and Ba, 2015) with an initial learning rate of 5e-3 and L2 weight decay of 5e-4. We use an early stopping strategy on the label accuracy of the validation set, with a patience of 20 epochs. We attempt to stack 0-3 ERNet layers and analyze the effect of different layer numbers.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
Besides traditional evaluation metrics such as label accuracy and F1, we use other two metrics to evaluate our model.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
FEVER score. The FEVER score is the label accuracy conditioned on providing at least one complete set of evidence. Claims labeled as "NEI" do not need the evidence.   OFEVER score. The document retrieval and sentence selection components are usually evaluated by the oracle FEVER (OFEVER) score, which is the upper bound of the FEVER score by assuming perfect downstream systems.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
For all of the experiments with GEAR, the scores (label accuracy, FEVER score) we report on the dev set are mean values with 10 runs initialized by different random seeds.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
In this section, we first present the evaluations of the document retrieval and sentence selection components. Then we evaluate our GEAR framework in several different aspects. Finally, we present a case study to demonstrate the effectiveness of our framework.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
We use the OFEVER metric to evaluate the document retrieval component. Table 3 shows the OFEVER scores of our model and models from other teams. After running the same model proposed by Hanselowski et al. (2018), we find our OFEVER score is slightly lower, which may due to the random factors.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
Then we compare our sentence selection component with different thresholds, as shown in Table 4. We find the model with threshold 0 achieves the highest recall and OFEVER score. When the threshold increases, the recall value and the OFEVER score drop gradually while the precision and F1 score increase. The results are consistent with our intuition. If we do not filter out evidence, more claims could be provided with the full evidence set. If we increase the value of the threshold, more pieces of noisy evidence are filtered out, which contributes to the increase of precision and F1.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
In this section, we evaluate our GEAR framework in different aspects. We first compare the label accuracy scores between our framework and baseline systems. Then we explore the effect of different thresholds from the upstream sentence filter. We also conduct additional experiments to check the effect of sentence embedding. As there are nearly 39% of claims require reasoning over multiple pieces of evidence, we construct a difficult dev subset and check the effectiveness of our ER-Net for evidence reasoning. Finally, we make an error analysis and provide the theoretical upperbound label accuracy of our framework.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
We use the label accuracy metric to evaluate the effectiveness of different claim verification models. The second column of Table 7 shows the label accuracy of different models on the dev set. We find the BERT fine-tuning models outperform all of the models from the shared task, which shows the strong capacity of BERT in representation learning and semantic understanding. The BERT-Concat model has a slight improvement over BERT-Pair, which is 0.37%.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
Our final model outperforms the best BERT-Concat baseline by 1.17%. As our framework provides a better way for evidence aggregating and reasoning, the improvement demonstrates that our framework has a better ability to integrate features from different evidence by propagating, analyzing and aggregating the features.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
The rightmost column of Table 4 shows the results of our GEAR frameworks with different sentence selection thresholds. We choose the model with threshold τ = 10 −3 , which has the highest label accuracy, as our final model. When the threshold increases from 0 to 10 −3 , the label accuracy increases due to less noisy information. However, when the threshold increases from 10 −3 to 10 −1 ,   the label accuracy decreases because informative evidence is filtered out, and the model can not obtain sufficient evidence to make the right inference.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
The BERT model we used in the sentence encoding step is fine-tuned on the FEVER dataset for one epoch. We need to find out whether the finetuning process or simply incorporating the sentence embeddings from BERT makes the major contribution to the final result. We conduct an experiment using a BERT model without the finetuning process and we find the final dev label accuracy is close to the result from a random guess. Therefore, the fine-tuning process rather than sentence embeddings plays an important role in this task. We need the fine-tuning process to capture the semantic and logical relations between evidence and the claim. Sentence embeddings are more general and cannot perform well in this specific task. So that we cannot just use sentence embeddings from other methods (e.g., ELMo, CNN) to replace the sentence embeddings we used here.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
In our observation, more than half of the claims in the dev dataset only need one piece of evidence to make the right inference. To verify the effectiveness of our framework on reasoning over multiple pieces of evidence, we build a difficult dev sub-  We test our final model on the difficult subset and present the results in Table 5. We find our models with ERNet perform better than models without ERNet and the minimal improvement between them is 1.27%. We can also discover from the table that models with 2 ERNet layers achieve the best results, which indicates that claims from the difficult subset require multi-step evidence propagation. This result demonstrates the ability of our framework to deal with claims which need multiple evidence.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
In this section, we examine the effect of errors propagating from upstream components. We utilize an evidence-enhanced dev subset, which assumes all pieces of ground truth evidence are retrieved, to test the theoretical upper-bound score of our GEAR framework.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
In our analysis, the main errors of our framework come from the upstream document retrieval and sentence selection components which can not extract sufficient evidence for inferring. For example, to verify the claim "Giada at Home was only available on DVD", we need the evidence "Giada at Home is a television show and first aired on October 18, 2008, on the Food Network." and "Food Network is an American basic cable and satellite television channel.". However, the entity linking Claim: Al Jardine is an American rhythm guitarist. (1) He is best known as the band's rhythm guitarist, and for occasionally singing lead vocals on singles such as "Help Me, Rhonda" (1965), "Then I Kissed Her" (1965) and "Come Go with Me" (1978).
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
(2) Alan Charles Jardine (born September 3, 1942) is an American musician, singer and songwriter who co-founded the Beach Boys.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
(3) In 2010, Jardine released his debut solo studio album, A Postcard from California. (4) In 1988, Jardine was inducted into the Rock and Roll Hall of Fame as a member of the Beach Boys. (5) Ray Jardine American rock climber, lightweight backpacker, inventor, author and global adventurer.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
Label: SUPPORTED Table 8: A case of the claim that requires integrating multiple evidence to verify. The representation for evidence "{DocName, LineNum}" means the evidence is extracted from the document "DocName" and of which the line number is LineNum. method used in our document retrieval component could not retrieve the "Food Network" document only from parsing the content of the claim. Thus the claim verification component can not make the right inference with insufficient evidence.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
To explore the effect of this issue, we test our models on an evidence-enhanced dev set, in which we add the ground truth evidence with relevance score 1 into the evidence set before the sentence threshold filter. It ensures that each claim in the evidence-enhanced set is provided with the ground truth evidence as well as the retrieved evidence.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
The experimental results are shown in Table 6. We can find that all scores in the table increase by more than 1.4% compared to the original dev set label accuracy in Table 7 because of the addition of the ground truth evidence. Because of the assumption of oracle upstream components, the results in Table 6 indicate the theoretical upper bound label accuracy of our framework.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
The results show the challenges in the previous evidence retrieval task, which could not be solved by current models. Nie et al. (2019) propose a two-hop evidence enhancement method which improves 0.08% on their final FEVER score. As the addition of the ground truth evidence leads to a   Table 8. The first five rows indicate the attention weights from nodes 1 to 5 in the first ERNet layer and the last row shows the attention weights from the attention aggregator. more than 1.4% increase in our experiment, it is worthwhile to design a better evidence retrieval pipeline, which remains to be our future research.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
We present the evaluation of our full pipeline in this section. Note that there is a gap between the label accuracy and the final FEVER score due to the completeness of the evidence set. We find that a model which is good at predicting NEI instances tends to obtain higher FEVER score. So we choose our final model based on the dev FEVER score among all of our experiments. This model contains one layer of ERNet and uses the attention aggregator. The threshold of the sentence filter is 10 −3 . Table 7 presents the evaluations of the full pipeline. We find the test FEVER score of BERT fine-tuning systems outperform other shared task models by nearly 1%. Furthermore, our full pipeline outperforms the BERT-Concat baseline by 1.46% and achieves significant improvements. Table 8 shows an example in our experiments which needs multiple pieces of evidence to make the right inference. The ground truth evidence set contains the sentences from the article "Al Jardine" with line number 0 and 1. These two pieces of evidence are also ranked at top two in our retrieved evidence set. To verify whether "Al Jardine is an American rhythm guitarist", our model needs the evidence "He is best known as the bands rhythm guitarist" as well as the evidence "Alan Charles Jardine ... is an American musician". We plot the attention map from our final model with one layer of ERNet and the attention aggregator in Figure 2. We can find that all evidence nodes tend to attend the first and the second evidence nodes, which provide the most useful information in this case. The attention weights in other evidence nodes are pretty low, which indicates that our model has the ability to select useful information from multiple pieces of evidence.
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification
We propose a novel Graph-based Evidence Aggregating and Reasoning (GEAR) framework on the claim verification subtask of FEVER. The framework utilizes the BERT sentence encoder, the evidence reasoning network (ERNet) and an evidence aggregator to encode, propagate and aggregate information from multiple pieces of evidence. The framework is proven to be effective and our final pipeline achieves significant improvements. In the future, we would like to design a multi-step evidence extractor and incorporate external knowledge into our framework.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Open-domain question answering (QA) has been a long standing problem in natural language understanding, information retrieval, and related fields . An typical open-domain QA system follows the retrieval-reader framework (Chen et al., 2017;Guu et al., 2020;, where the relevant passages are first retrieved from a large text corpus, and a reader module then navigates multiple passages for answer inference. In this work, we study two paradigms of reader modules, i.e. extractive Guu et al., 2020) and generative Izacard and Grave, 2021) readers. The extractive reader extracts contiguous spans from the retrieved passages whereas the generative reader sequentially decodes the answer string which might not be contained in the retrieved passages. * Equal Contribution
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Open-domain question answering (QA) has been a long standing problem in natural language understanding, information retrieval, and related fields . An typical open-domain QA system follows the retrieval-reader framework (Chen et al., 2017;Guu et al., 2020;, where the relevant passages are first retrieved from a large text corpus, and a reader module then navigates multiple passages for answer inference. In this work, we study two paradigms of reader modules, i.e. extractive Guu et al., 2020) and generative Izacard and Grave, 2021) readers. The extractive reader extracts contiguous spans from the retrieved passages whereas the generative reader sequentially decodes the answer string which might not be contained in the retrieved passages. * Equal Contribution
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Recent work on open-domain QA Guu et al., 2020;Izacard and Grave, 2021) explores either an extractive reader or a generative reader exclusively. We hypothesize that extractive and generative readers adopt different answer inference strategies, thus a hybrid extractive/generative reader can be a better option for open-domain QA tasks. As shown in Figure 1, compared with prediction agreement among only generative or extractive readers (topleft and bottom-right), the cross prediction agreement between extractive and generative readers (bottom-left) is relatively low (<50%). It indicates that answers produced by those two types of models are different and they can be complementary to each other. Therefore, we propose a hybrid reader approach, UnitedQA, which is a simple ensemble approach to combine the predictions from extractive and generative readers. It achieves state-of-theart results on NaturalQuestions (Kwiatkowski et al., 2019) and TriviaQA (Joshi et al., 2017).
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Recent work on open-domain QA Guu et al., 2020;Izacard and Grave, 2021) explores either an extractive reader or a generative reader exclusively. We hypothesize that extractive and generative readers adopt different answer inference strategies, thus a hybrid extractive/generative reader can be a better option for open-domain QA tasks. As shown in Figure 1, compared with prediction agreement among only generative or extractive readers (topleft and bottom-right), the cross prediction agreement between extractive and generative readers (bottom-left) is relatively low (<50%). It indicates that answers produced by those two types of models are different and they can be complementary to each other. Therefore, we propose a hybrid reader approach, UnitedQA, which is a simple ensemble approach to combine the predictions from extractive and generative readers. It achieves state-of-theart results on NaturalQuestions (Kwiatkowski et al., 2019) and TriviaQA (Joshi et al., 2017).
UnitedQA: A Hybrid Approach for Open Domain Question Answering
In UnitedQA, the extractive reader (UnitedQA-E) and generative reader (UnitedQA-G) are built upon the pretrained language models, ELECTRA (Clark et al., 2020) and T5 (Raffel et al., 2020), respectively. For the UnitedQA-E, we adopt a weakly-supervised training objective to address the noisy supervision issue caused by the heuristicsbased labeling and incorporate the posterior differential regularization (PDR)  to improve the model robustness. The UnitedQA-G follows the T5 Fusion-in-Decoder (FID) (Izacard and Grave, 2021) and we make two improvements: first, we add a group of attention bias parameters into the decoder cross-attention block to feature the ranking information of retrieved contexts; second, we add the adversarial training (Ju et al., 2019;Jiang et al., 2020; to improve the model generalization ability.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
In UnitedQA, the extractive reader (UnitedQA-E) and generative reader (UnitedQA-G) are built upon the pretrained language models, ELECTRA (Clark et al., 2020) and T5 (Raffel et al., 2020), respectively. For the UnitedQA-E, we adopt a weakly-supervised training objective to address the noisy supervision issue caused by the heuristicsbased labeling and incorporate the posterior differential regularization (PDR)  to improve the model robustness. The UnitedQA-G follows the T5 Fusion-in-Decoder (FID) (Izacard and Grave, 2021) and we make two improvements: first, we add a group of attention bias parameters into the decoder cross-attention block to feature the ranking information of retrieved contexts; second, we add the adversarial training (Ju et al., 2019;Jiang et al., 2020; to improve the model generalization ability.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
The experimental results highlight the effec-   (Kwiatkowski et al., 2019) and TriviaQA (Joshi et al., 2017), respectively. It is worth noting that our UnitedQA model not only outperforms each single model but also brings more pronounced improvements over homogeneous ensembles of either extractive or generative readers. Last, based on our analyses, UnitedQA-E and UnitedQA-G have advantages in different cases, suggesting they may use different reasoning strategies.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
The experimental results highlight the effec-   (Kwiatkowski et al., 2019) and TriviaQA (Joshi et al., 2017), respectively. It is worth noting that our UnitedQA model not only outperforms each single model but also brings more pronounced improvements over homogeneous ensembles of either extractive or generative readers. Last, based on our analyses, UnitedQA-E and UnitedQA-G have advantages in different cases, suggesting they may use different reasoning strategies.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
In this section, we present the overall pipeline of the UnitedQA system, which consists of three components: Retrieval, Reading, and Re-ranking. First, the retrieval module fetches a list of relevant passages from a Wikipedia dump for a given question. Then, the module of hybrid readers produces answer candidates from the set of retrieved passages. Last, the re-ranking module combines the answer candidates with linear interpolation and produce the final answer.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Retrieval Following , we consider two methods, BM25 and dense passage retrieval (DPR), for retrieving the support passages for a given question. For BM25, passages are encoded as bag of words (BOW), and inverse document frequencies are used as the ranking function. For DPR, passages and questions are represented as dense vectors based on two BERT  models. The relevance score is then computed based on the dot production between the query and passage vectors. In this paper, we adopt the same implementation as  for retrieving passages. Specifically, the English Wikipedia dump from Dec. 20, 2018 is used as the source documents for retrieval, with the removal of semi-structured data, such as tables or lists. Each document is split into disjoint 100-word passages as the basic retrieval unit. The top-100 passages are then passed for reading.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Reading We combine the generative reader and the extractive reader to produce answer candidates over the retrieved passages. Here, we only give a highlevel description of our approach. More details regarding our improved extractive and generative models are presented in §2.1 and §2.2 respectively. The generative reader is based on a sequenceto-sequence model pre-trained in a forwardgeneration fashion on a large corpus, i.e. T5 (Raffel et al., 2020). Similar to Izacard and Grave (2021), the model takes the question and its relevant passages as input, and then generates the answer string token by token. Specifically, the concatenation of all retrieved passages and the corresponding question is used as the encoder input. Then, the decoder performs reasoning over the concatenation of all evidence through an attention mechanism.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Reading We combine the generative reader and the extractive reader to produce answer candidates over the retrieved passages. Here, we only give a highlevel description of our approach. More details regarding our improved extractive and generative models are presented in §2.1 and §2.2 respectively. The generative reader is based on a sequenceto-sequence model pre-trained in a forwardgeneration fashion on a large corpus, i.e. T5 (Raffel et al., 2020). Similar to Izacard and Grave (2021), the model takes the question and its relevant passages as input, and then generates the answer string token by token. Specifically, the concatenation of all retrieved passages and the corresponding question is used as the encoder input. Then, the decoder performs reasoning over the concatenation of all evidence through an attention mechanism.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Following state-of-the-art extractive QA models , our extractive reader is based on a Transformer neural network pre-trained with a cloze style selfsupervised objective, i.e. ELECTRA (Clark et al., 2020). Here, a pair of a given question and a support passage is jointly encoded into neural text representations. These representations are then used to define scores or probabilities of possible answer begin and end positions, which are in turn used to define probabilities over possible answer spans.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Following state-of-the-art extractive QA models , our extractive reader is based on a Transformer neural network pre-trained with a cloze style selfsupervised objective, i.e. ELECTRA (Clark et al., 2020). Here, a pair of a given question and a support passage is jointly encoded into neural text representations. These representations are then used to define scores or probabilities of possible answer begin and end positions, which are in turn used to define probabilities over possible answer spans.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Finally, the answer string probabilities are based on the aggregation over all possible answer spans from the entire set of support passages.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
In §2.1.2, we give the problem definition of opendomain QA for extractive reader. Then, we detail the improvements of UnitedQA-E in §2.1.2.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Given a question q and a set of K retrieved passages p 1 , . . . , p K , a text encoder produces contextualized representations: h k 1 , ...h k T ∈ R n for the question-passage pair (q, p k ) in the form of
UnitedQA: A Hybrid Approach for Open Domain Question Answering
and [SEP]are special tokens for encoding inputs, T is the maximum sequence length of the input text, and h k i indicates the contextualized embedding of the i-th token in (q, p k ).
UnitedQA: A Hybrid Approach for Open Domain Question Answering
The extractive reader computes the span-begin score of the i-th token as s b (i k ) = w T b h k i using a weight vector w b ∈ R d . The span-end score s e (j k ) is defined in the same way. Thus, the probabilities of a start position i k and an end position
UnitedQA: A Hybrid Approach for Open Domain Question Answering
where Z b , Z e are normalizing factors defined by the corresponding probability space. The probability of an answer span from i k to j k is defined as
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Here, we consider two probability spaces, passage level and multi-passage level, with the only difference in the computing of Z b , Z e . Specifically, the passage-level probability of each answer begin and end is computed by normalizing all possible positions in the respective passage, i.e.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
, Z e = Z k e = I k ∪NULL exp(s e (j)), where I k is the set of all possible positions from the k-th passage and NULL indicates special positions if p k does not support answering the question. Similarly, the multi-passage level probability is computed by normalizing over each answer positions across all K relevant pas-
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Since there are usually multiple plausible mentions for open-domain QA, during training, it is typical to maximize either the marginal log-likelihood (MML) of all correct spans  or the log-likelihood of the most likely correct span (HardEM) (Min et al., 2019). During inference, the prediction is made based on the candidate answer string score, obtaining as P a (y) = (i,j)∈Y P s (i, j), where Y is the set of spans corresponding to the answer string y.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Since there are usually multiple plausible mentions for open-domain QA, during training, it is typical to maximize either the marginal log-likelihood (MML) of all correct spans  or the log-likelihood of the most likely correct span (HardEM) (Min et al., 2019). During inference, the prediction is made based on the candidate answer string score, obtaining as P a (y) = (i,j)∈Y P s (i, j), where Y is the set of spans corresponding to the answer string y.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
In addition to better text representations from Clark et al. (2020), we consider two methods for improving the training of the extractive reader.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
In addition to better text representations from Clark et al. (2020), we consider two methods for improving the training of the extractive reader.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Multi-objective for Weakly-supervised QA The multi-objective formulation is introduced in Cheng et al. (2020) for improving weakly supervised document-level QA. Different from Cheng et al. (2020) where only MML is considered for the multi-objective formulation, we found combining HardEM with MML is more effective for open-domain QA based on our experiments ( §4.1). Specifically, we combine a multi-passage HardEM loss with K passage-level MML losses over a batch of K passages
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Multi-objective for Weakly-supervised QA The multi-objective formulation is introduced in Cheng et al. (2020) for improving weakly supervised document-level QA. Different from Cheng et al. (2020) where only MML is considered for the multi-objective formulation, we found combining HardEM with MML is more effective for open-domain QA based on our experiments ( §4.1). Specifically, we combine a multi-passage HardEM loss with K passage-level MML losses over a batch of K passages
UnitedQA: A Hybrid Approach for Open Domain Question Answering
where P M s , P P s is the multi-passage level and passage level span probabilities respectively. Posterior Differential Regularization Due to the noisy supervision for open-domain QA (Chen et al., 2017), we investigate the posterior differential regularization (PDR)  to improve the robustness of the extractive reader. Different from  where only clean supervision setting is considered, in this work, we apply PDR to the weakly supervised open-domain QA scenario. Given it is computationally expensive to enumerate all possible spans, we apply two separate regularization terms for the begin and end probabilities at the multi-passage level, respectively,
UnitedQA: A Hybrid Approach for Open Domain Question Answering
where P M s , P P s is the multi-passage level and passage level span probabilities respectively. Posterior Differential Regularization Due to the noisy supervision for open-domain QA (Chen et al., 2017), we investigate the posterior differential regularization (PDR)  to improve the robustness of the extractive reader. Different from  where only clean supervision setting is considered, in this work, we apply PDR to the weakly supervised open-domain QA scenario. Given it is computationally expensive to enumerate all possible spans, we apply two separate regularization terms for the begin and end probabilities at the multi-passage level, respectively,
UnitedQA: A Hybrid Approach for Open Domain Question Answering
where D(·|·) is the squared Hellinger distance, and P b , P e are the probabilities of start and end positions with additive input noise to the token embeddings. Specifically, we sample noise vectors 1 , . . . , T from N (0, c 2 I), and add them to the token embeddings as the noisy input, i.e. v 1 + 1 , . . . , v T + T , where c is fixed to 1e−3 throughout our experiments.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Based on this, the overall training objective for the extractive reader is
UnitedQA: A Hybrid Approach for Open Domain Question Answering
where γ is a regularization scalar hyperparameter.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Here, we first formally define the setup of generative reader for open-domain QA in § 2.2.1 and then present our improvements in § 2.2.2.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Given a question q and a set of K retrieved passages p 1 , . . . , p K , the encoder model encodes each (q, p k ) pair independently, and produces contextualized representation for each token: h k i ∈ R d for the i-th token of the k-th pair. The decoder then performs attention over the concatenation of the representations of all the retrieved passages, and generates the answer string.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Let x denote the input of the question and all retrieved passages x = (q, p 1 ), ..., (q, p K ) , and y the answer string with its tokens as (y 1 , ..., y N ). The generative reader is trained to maximize a sequence-to-sequence objective for a given (x, y),
UnitedQA: A Hybrid Approach for Open Domain Question Answering
where θ is the model parameter. During inference, a greedy decoding is used to produce the answer.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
The decoder in the T5 transformer model adopts a cross-attention mechanism to compute attention scores between the decoding answer tokens and all the retrieved passage tokens. Specifically, let y i ∈ R d be the query vector of the i-th decoding token 1 , and m k j ∈ R d be the key vector of the j-th token in ((q), p k ). The multi-head cross-attention scores in T5 (Raffel et al., 2020) 
UnitedQA: A Hybrid Approach for Open Domain Question Answering
The decoder in the T5 transformer model adopts a cross-attention mechanism to compute attention scores between the decoding answer tokens and all the retrieved passage tokens. Specifically, let y i ∈ R d be the query vector of the i-th decoding token 1 , and m k j ∈ R d be the key vector of the j-th token in ((q), p k ). The multi-head cross-attention scores in T5 (Raffel et al., 2020) 
UnitedQA: A Hybrid Approach for Open Domain Question Answering
where |Head| is the number of attention heads. However, it doesn't capture the relevance information of retrieved passages into the reader in (5). To add the relevance feature into the attention block, we revise (5) by incorporating the attention bias
UnitedQA: A Hybrid Approach for Open Domain Question Answering
where b k ∈ R |Head| is a trainable attention bias vector for all the tokens in the k-th retrieved passage. In the experiments, the maximum retrieved passages is by default set to 100. Thus, the decoder attention bias introduces additional 100 * |Head| parameters for each layer. Adversarial Training Adversarial training creates adversarial examples by adding small perturbations to the embedding layer. Assuming the word(piece) embedding layer is parameterized by a matrix V ∈ R |V |×d , |V | is the vocabulary size, and d 1 we omit the layer notation for simplification is the embed-dimension. The adversarial embedding matrixV can be obtained by
UnitedQA: A Hybrid Approach for Open Domain Question Answering
where SG(·) is the stop-gradient operation. We use the adversarial embedding matrixV to replace the original V in model parameters θ, and obtainθ. Thus the adversarial loss can be calculated as
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Therefore, the overall training objective of the generative reader is
UnitedQA: A Hybrid Approach for Open Domain Question Answering
where α = 0.5, β = 0.5 in all of the exepriments.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
The UnitedQA system combines outputs from both extractive and generative models for a given question during inference. Since the output spaces of extractive and generative models are different, we use a simple linear interpolation based on best predictions from each model 2 . Denote the predicted strings from M extractive and N generative models as y E 1 , ..., y E M and y G 1 , ..., y G N , respectively. The hybrid prediction y * is obtained by
UnitedQA: A Hybrid Approach for Open Domain Question Answering
where Y is the set of all predicted strings, 1(y, y ) is an indicator function and τ = 0.6, δ = 0.4.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
We use two representative QA datasets and adopt the same training/dev/testing splits as in previous  work . Both datasets (see Table 1 for statistics) have been heavily studied in recent work Min et al., 2019;Guu et al., 2020). We follow the standard evaluation protocol and use exact match (EM) as the evaluation metric.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
We use two representative QA datasets and adopt the same training/dev/testing splits as in previous  work . Both datasets (see Table 1 for statistics) have been heavily studied in recent work Min et al., 2019;Guu et al., 2020). We follow the standard evaluation protocol and use exact match (EM) as the evaluation metric.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
NaturalQuestions ( i.e. answers with less than 5 tokens. In the NaturualQuestions, the questions are considered to be more information seeking given that the question askers didn't know the answer beforehand. In addition, we use another evaluation set, i.e. the dev set introduced recently by the EfficientQA competition (Min et al., 2021), which is constructed in the same way as the original NaturalQuestions dataset.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
TriviaQA (Joshi et al., 2017) contains trivia question-answer pairs that were scraped from the web. Different from NaturalQuestions, the questions here are written with known answers in mind. Specifically, the unfiltered set has been used for developing open-domain QA models.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
TriviaQA (Joshi et al., 2017) contains trivia question-answer pairs that were scraped from the web. Different from NaturalQuestions, the questions here are written with known answers in mind. Specifically, the unfiltered set has been used for developing open-domain QA models.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Implementation details For a fair comparison, we use the same retrieval module as  for NaturalQuestions and TriviaQA to mitigate the impact of retrieval difference. Specifically, we use DPR (single) for NaturalQuestions and BM25+DPR (multi) for TriviaQA because of their best end-to-end performance . For all the experiments, we use 8 and 16 V100-32GB for base and large model training respectively. We train our models with Adam optimizer of a linear scheduler with a warmup raito of 0.1. The extractive models are trained for up to 8 epochs with a learning rate of 2e−5 and a batch passage size per question of 16. The generative models are trained for up to 10 epochs with a learning rate of 1e−4, a batch size of 64, and 100 retrieved passages per question for model training.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
We select γ in {4, 8}. After the best configuration is selected based on the dev set, we run our best models 3 times independently with different random seeds and report the median performance on the test set. We also report ensemble results which are based on the linear interpolation over answer predictions from the 3 models.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Single Model Results: We first compare our models to two recent models, REALM (Guu et al., 2020) and RAG , which are first pre-trained with different retrieval augmented objectives and then fine-tuned for open-domain QA. In addition, we include as baselines DPR  and T5-FID (Izacard and Grave, 2021), both of which are based on the same retriever as ours. As shown in Table 2, both our extractive and generative models achieve new stateof-the-art results for both studied datasets. Compared with the recent state-of-the-art extractive model (DPR), our base model leads to pronounced 15% relative improvements for both NaturalQuestions (+6.2 absolute improvement) and TriviaQA (+8.4 absolute improvement). More importantly, UnitedQA-E base achieves comparable or even better performance with regard to generative models of larger size, i.e. RAG and T5-FID base . It highlights the importance of proper training strategies for open-domain QA models.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Single Model Results: We first compare our models to two recent models, REALM (Guu et al., 2020) and RAG , which are first pre-trained with different retrieval augmented objectives and then fine-tuned for open-domain QA. In addition, we include as baselines DPR  and T5-FID (Izacard and Grave, 2021), both of which are based on the same retriever as ours. As shown in Table 2, both our extractive and generative models achieve new stateof-the-art results for both studied datasets. Compared with the recent state-of-the-art extractive model (DPR), our base model leads to pronounced 15% relative improvements for both NaturalQuestions (+6.2 absolute improvement) and TriviaQA (+8.4 absolute improvement). More importantly, UnitedQA-E base achieves comparable or even better performance with regard to generative models of larger size, i.e. RAG and T5-FID base . It highlights the importance of proper training strategies for open-domain QA models.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Hybrid Model Results: In order to evaluate the advantage of the hybrid of the extractive and generative models (UnitedQA), we include two homogeneous ensemble baselines, one consisting of only extractive readers (UnitedQA-E++) and the other ensemble of exclusively generative models (UnitedQA-G++). For homogeneous ensemble cases, the three-way majority prediction is used. For the hybrid of extractive and generative readers, we select a three-model combination from the set of three generative and three extractive models based on the dev set. We observed that combining predictions from two generative models and one extractive model results in the best hybrid model for both datasets. As expected, all ensemble models show an improvement over their single model counterparts. However, the two homogeneous ensemble baselines, UnitedQA-E++ and UnitedQA-G++, only provide marginal gains over the corresponding best single models. The significant improvement brought by our proposed hybrid approach indicates the benefit of combining extractive and generative readers for open-domain QA.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Discussion: Although the proposed hybrid approach has been shown to be highly effective for open-domain QA, we point out that the improved performance comes with increased computational cost. The best combination requires approximately three times the computational cost of a single generative model. Therefore, it would be interesting to explore more efficient hybrid methods, such as effective parameter sharing strategies or unified formulations. Another interesting future direction is to explore customized compression approaches for reducing the model size of retriever and reader separately or jointly through pruning (Han et al., 2016), quantization (Hubara et al., 2018), and knowledge distillation (Hinton et al., 2015). Specifically, given that the hybrid model is more effective, it is likely that a student model can learn more effectively from a hybrid teacher model via knowledge distillation for open-domain QA.  
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Discussion: Although the proposed hybrid approach has been shown to be highly effective for open-domain QA, we point out that the improved performance comes with increased computational cost. The best combination requires approximately three times the computational cost of a single generative model. Therefore, it would be interesting to explore more efficient hybrid methods, such as effective parameter sharing strategies or unified formulations. Another interesting future direction is to explore customized compression approaches for reducing the model size of retriever and reader separately or jointly through pruning (Han et al., 2016), quantization (Hubara et al., 2018), and knowledge distillation (Hinton et al., 2015). Specifically, given that the hybrid model is more effective, it is likely that a student model can learn more effectively from a hybrid teacher model via knowledge distillation for open-domain QA.  
UnitedQA: A Hybrid Approach for Open Domain Question Answering
In this section, we first carry out ablation study on the extractive and generative model improvements. Moreover, we aim to take a deeper look and understand the difference between the two models.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
In Table 3, we present ablation experiments on the effectiveness of different textual representations and methods for improving the extractive model UnitedQA-E base . Here, we focus on base models, i.e. BERT base and ELECTRA base . Note that the row UnitedQA-E base is the corresponding base model reported in Table 2. Compared with the MML-based multi-objective (Cheng et al., 2020), we find that a new multi-objective with HardEM at the multi-passage level and MML at the passage level is more effective for open-domain QA. In addition to the multi-objective training, there is a noticeable improvement brought by the regularization method (PDR) which indicates the importance of proper regularization for learning with noisy supervision. Last but not least, the large improvement of ELECTRA over BERT indicates the importance of deriving better text representations for weakly supervised NLP problems. For the UnitedQA-G, we present the ablation study on analyzing the effectiveness of decoder attention bias component and adversarial training mechanism in Table 4. Both techniques contribute to decent improvements over T5-FID with more pronounced gains brought by adversarial training.   
UnitedQA: A Hybrid Approach for Open Domain Question Answering
In Table 3, we present ablation experiments on the effectiveness of different textual representations and methods for improving the extractive model UnitedQA-E base . Here, we focus on base models, i.e. BERT base and ELECTRA base . Note that the row UnitedQA-E base is the corresponding base model reported in Table 2. Compared with the MML-based multi-objective (Cheng et al., 2020), we find that a new multi-objective with HardEM at the multi-passage level and MML at the passage level is more effective for open-domain QA. In addition to the multi-objective training, there is a noticeable improvement brought by the regularization method (PDR) which indicates the importance of proper regularization for learning with noisy supervision. Last but not least, the large improvement of ELECTRA over BERT indicates the importance of deriving better text representations for weakly supervised NLP problems. For the UnitedQA-G, we present the ablation study on analyzing the effectiveness of decoder attention bias component and adversarial training mechanism in Table 4. Both techniques contribute to decent improvements over T5-FID with more pronounced gains brought by adversarial training.   
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Here, we vary the number of retrieved passages during inference and report the evaluation results in terms of end-to-end QA exact match score of UnitedQA-E and UnitedQA-G along with the corresponding top-k retrieval accuracy. The results are summarized in Table 5. As expected, when the number of retrieved passages increases, both top-k retrieval accuracy and the end-to-end QA performance improve. However, there is a noticeable gap between the improvement of retrieving more passages (i.e., recall) and that of the corresponding end-to-end QA performance, especially for the extractive reader. This is likely caused by additional noise introduced with improved retrieval recall. Specifically, only half of the retriever improvement can be effectively utilized by the extractive model while the generative model can benefit more from retrieving more passages. This suggests that by concatenating all passages in vector space, the generative model are more effective in de-noising in comparison to the extractive model.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Following Lewis et al. (2021), we carry out a breakdown evaluation of model performance over the NaturalQuestions and TriviaQA test sets. Given their superior performance, we again only consider our improved extractive and generative models, i.e. UnitedQA-E large and UnitedQA-G respectively. The evaluation is summarized in Table 6. In comparison to their corresponding overall performance, both the extractive and generative models achieve much better performance on the "Overlap" categories (i.e. "Question Overlap" and "Answer Overlap") for both NaturalQuestions and TrivaQA, which indicates that both models perform well for question and answer memorization. Different from question and answer memorization, there is a pronounced performance drop for both models on the"Answer Overlap Only" category where certain amount of relevance inference capability is required to succeed. Lastly, we see that both extractive and generative models suffer some significant performance degradation for the "No Overlap" column which highlights model's generalization evaluation. Nevertheless, the extractive model demonstrate a better QA generalization by achieving a better overall performance on the "No Overlap" category for both datasets.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Following Lewis et al. (2021), we carry out a breakdown evaluation of model performance over the NaturalQuestions and TriviaQA test sets. Given their superior performance, we again only consider our improved extractive and generative models, i.e. UnitedQA-E large and UnitedQA-G respectively. The evaluation is summarized in Table 6. In comparison to their corresponding overall performance, both the extractive and generative models achieve much better performance on the "Overlap" categories (i.e. "Question Overlap" and "Answer Overlap") for both NaturalQuestions and TrivaQA, which indicates that both models perform well for question and answer memorization. Different from question and answer memorization, there is a pronounced performance drop for both models on the"Answer Overlap Only" category where certain amount of relevance inference capability is required to succeed. Lastly, we see that both extractive and generative models suffer some significant performance degradation for the "No Overlap" column which highlights model's generalization evaluation. Nevertheless, the extractive model demonstrate a better QA generalization by achieving a better overall performance on the "No Overlap" category for both datasets.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Here, we conduct analyses into prediction errors made by the extractive and generative models based on automatic evaluation. For this study, we use the EfficientQA dev set (Min et al., 2021) which is constructed in the same way as the original Natu-ralQuestions dataset. Specifically, we group prediction errors into three categorizes: 1) common prediction errors made by both the extractive and generative models, 2) prediction errors made by the extractive model, 3) prediction errors produced by the generative model. In the following, we first carry out a manual inspection into the common errors. Then, we compare the prediction errors made by extractive and generative models, respectively. First of all, there is an error rate of 29% of those consensus predictions made by both extractive and generative models according to the automatic evaluation. Based on 30 randomly selected examples, we find that around 30% of those predictions are actually valid answers as shown in the top part of Table 7. In addition to predictions that are answers at different granularity or semantically equivalent ones, some of those prediction errors are likely caused by the ambiguity in questions. As the given example in Table 7, based on the specificity, the model prediction is also a valid answer. This high-  Table 6: Breakdown evaluation on NaturalQuestions (NQ) and TriviaQA based on test splits defined in (Lewis et al., 2021). Exact match scores are reported. UnitedQA-E and UnitedQA-G denote our extractive and generative models respectively.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Here, we conduct analyses into prediction errors made by the extractive and generative models based on automatic evaluation. For this study, we use the EfficientQA dev set (Min et al., 2021) which is constructed in the same way as the original Natu-ralQuestions dataset. Specifically, we group prediction errors into three categorizes: 1) common prediction errors made by both the extractive and generative models, 2) prediction errors made by the extractive model, 3) prediction errors produced by the generative model. In the following, we first carry out a manual inspection into the common errors. Then, we compare the prediction errors made by extractive and generative models, respectively. First of all, there is an error rate of 29% of those consensus predictions made by both extractive and generative models according to the automatic evaluation. Based on 30 randomly selected examples, we find that around 30% of those predictions are actually valid answers as shown in the top part of Table 7. In addition to predictions that are answers at different granularity or semantically equivalent ones, some of those prediction errors are likely caused by the ambiguity in questions. As the given example in Table 7, based on the specificity, the model prediction is also a valid answer. This high-  Table 6: Breakdown evaluation on NaturalQuestions (NQ) and TriviaQA based on test splits defined in (Lewis et al., 2021). Exact match scores are reported. UnitedQA-E and UnitedQA-G denote our extractive and generative models respectively.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Different  lights the limitation of the current evaluation metric, which does not accurately estimate the existing open-domain QA system capabilities. As shown in the bottom part of Table 7, most of representative errors are due to the confusion of related concepts, entities or events that are mentioned frequently together with the corresponding gold answers.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Next, all questions from the dev set are categorized based the WH question word, i.e. what, which, when, who, how, where. We then report the relative performance change of each WH category for both extractive and generative models over their corresponding overall prediction accuracy in Figure 2. First, it is easy to see that both extractive and generative models achieve the best performance for entity related who questions, which is likely to be the result of high ratio of samples of this type seen during training. In contrast, the answers to what questions can play a much richer syntactic role in context, making it more difficult for both extractive and generative models to perform well. Interestingly, the generative model exhibits the strength for temporal reasoning, whereas the extractive model does not. This difference suggests that it is worth exploring better temporal modeling strategies to improve the extractive model in the future.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
There has been a line of work on improving distantly-supervised reading comprehension models by developing learning methods and model architectures that can better use noisy labels. Most of them focus on the document-level QA, where all paragraphs share the same document context. Clark and Gardner (2018) propose a paragraphpair ranking objective for learning with multiple paragraphs so that the model can distinguish relevant paragraphs from irrelevant ones. In (Lin et al., 2018), a coarse-to-fine model is proposed to handle label noise by aggregating information from relevant paragraphs and then extracting answers from selected ones. Min et al. (2019) propose a hard EM learning scheme where only passage-level loss is considered for document-level QA. More recently, different probabilistic assumptions with corresponding training and inference methods are examined in (Cheng et al., 2020) again for documentlevel QA with distant supervision. In our work, we further extend the multi-objective formulation proposed in (Cheng et al., 2020) with the hard EM learning (Min et al., 2019) for enhancing extrac-tive open-domain QA, where the input passages are given by a retrieval model and are typically from different documents.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
There has been a line of work on improving distantly-supervised reading comprehension models by developing learning methods and model architectures that can better use noisy labels. Most of them focus on the document-level QA, where all paragraphs share the same document context. Clark and Gardner (2018) propose a paragraphpair ranking objective for learning with multiple paragraphs so that the model can distinguish relevant paragraphs from irrelevant ones. In (Lin et al., 2018), a coarse-to-fine model is proposed to handle label noise by aggregating information from relevant paragraphs and then extracting answers from selected ones. Min et al. (2019) propose a hard EM learning scheme where only passage-level loss is considered for document-level QA. More recently, different probabilistic assumptions with corresponding training and inference methods are examined in (Cheng et al., 2020) again for documentlevel QA with distant supervision. In our work, we further extend the multi-objective formulation proposed in (Cheng et al., 2020) with the hard EM learning (Min et al., 2019) for enhancing extrac-tive open-domain QA, where the input passages are given by a retrieval model and are typically from different documents.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
In this study, we propose a hybrid model for opendomain QA, called UnitedQA, which combines the strengths of extractive and generative readers. We demonstrate the effectiveness of UnitedQA on two popular open-domain QA benchmarks, Natu-ralQuestions and TriviaQA. Our results show that the proposed UnitedQA model significantly outperforms single extractive and generative models as well as their corresponding homogeneous ensembles, and sets new state-of-the-art on both benchmarks. We also perform a comprehensive empirical study to investigate the relative contributions of different components of our model and the techniques we use to improve the readers. For future work, it would be interesting to explore model compression approaches for reducing the model size of retriever and reader separately or jointly through pruning, quantization, and knowledge distillation.
KALA: Knowledge-Augmented Language Model Adaptation
Pre-trained Language Models (PLMs) (Devlin et al., 2019;Brown et al., 2020) have shown to be effective on various Natural Language Understanding (NLU) tasks. Although PLMs aim to address diverse downstream tasks from various data sources, there have been considerable efforts to adapt the PLMs to specific domains -distributions over the language characterizing a given topic or genre (Gururangan et al., 2020)-for which the acquisition of domain knowledge is required to accurately solve the downstream tasks (e.g., Biomedical Named Entity Recognition (Dogan et al., 2014)). * * Equal contribution This problem, known as Language Model Adaptation, can be viewed as a transfer learning problem (Yosinski et al., 2014;Ruder, 2019) under domain shift, where the model is pre-trained on the general domain and the labeled distribution is available for the target domain-specific task. The most prevalent approach to this problem is adaptive pre-training ( Figure 2a) which further updates all parameters of the PLM on a large domain-specific or curated task-specific corpus, with the same pretraining strategy (e.g., masked language modeling) before fine-tuning it on the downstream task (Beltagy et al., 2019;Lee et al., 2020;Gururangan et al., 2020). This continual pre-training of a PLM on the target domain corpus allows it to learn the distribution of the target domain, resulting in improved performance on domain-specific tasks (Howard and Ruder, 2018;Han and Eisenstein, 2019).
KALA: Knowledge-Augmented Language Model Adaptation
While it has shown to be effective, adaptive pretraining has obvious drawbacks. First, it is computationally inefficient. Although a PLM becomes more powerful with the increasing amount of pretraining data (Gururangan et al., 2020), further pre-training on the additional data requires larger memory and computational cost as the dataset size grows (Bai et al., 2021). Besides, it is difficult to adapt the PLM to a new domain without forgetting the general knowledge it obtained from the initial pretraining step, since all pre-trained parameters are continually updated to fit the domain-specific corpus during adaptive pre-training (Chen et al., 2020). This catastrophic forgetting of the task-Context: non-ST-elevation myocardial infarction, 100% RCA, three stents, 50% mid LD. 2017-04-02 instent restenosis status post brachytherapy. (…) her shortness of breath and asthma flare, tentracycline, sulfa, Demerol. MEDICATIONS (…) News News … Recently, sirolimus-eluting coronary stents have been shown to reduce restenosis and additional adverse cardiac events in patients with severe coronary artery disease (…) Domain Corpus (e.g. Medical Textbook) Question: Did the patient receive brachytherapies for instent restenosis? general knowledge may lead to the performance degradation on the downstream tasks. In Figure 1, we show that adaptive pre-training with more training steps could lead to performance degeneration. Thus, it would be preferable if we could adapt the PLM to the domain-specific task without costly adaptive pre-training. To this end, we aim to integrate the domain-specific knowledge into the PLM directly during the task-specific fine-tuning step, as shown in Figure 2b, eliminating the adaptive pre-training stage. Specifically, we first note that entities and relations are core building blocks of the domain-specific knowledge that are required to solve for the domain-specific downstream tasks. Clinical domain experts, for example, are familiar with medical terminologies and their complex relations. Then, to represent the domain knowledge consisting of entities and relations, we introduce the Entity Memory, which is the source of entity embeddings but independent of the PLM parameters (See Entity Memory in Figure 2b). Then, we further exploit the relational structures of the entities by utilizing a Knowledge Graph (KG), which denotes the factual relationships between entities, as shown in Knowledge Graph of Figure 2b.
KALA: Knowledge-Augmented Language Model Adaptation
The remaining step is how to integrate the knowledge into the PLM during fine-tuning. To this end, we propose a novel layer named Knowledgeconditioned Feature Modulation (KFM, §3.2), which scales and shifts the intermediate hidden representations of PLMs by conditioning them with retrieved knowledge representations. This knowledge integration scheme has several advantages. First, it does not modify the original PLM architecture, and thus could be integrated into any PLMs regardless of their architectures. Also, it only re-quires marginal computational and memory overhead, while eliminating the need of excessive further pre-training ( Figure 1). Finally, it can effectively handle unseen entities with relational knowledge from the KG, which are suboptimally embedded by adaptive pre-training. For example, as shown in Figure 2, an entity restenosis does not appear in the training dataset for fine-tuning, thus adaptive pre-training only implicitly infers them within the context from the broad domain corpus. However, we can explicitly represent the unknown entity by aggregating the representations of known entities in the entity memory (i.e., in Figure 2, neighboring entities, such as asthma and pethidine, are used to represent the unseen entity restenosis).
KALA: Knowledge-Augmented Language Model Adaptation
We combine all the previously described components into a novel language model adaptation framework, coined as Knowledge-Augmented Language model Adaptation (KALA) (Figure 3). We empirically verify that KALA improves the performance of the PLM over adaptive pre-training on various domains with two knowledge-intensive tasks: Question Answering (QA) and Named Entity Recognition (NER). Our contribution is threefold:
KALA: Knowledge-Augmented Language Model Adaptation
• We propose a novel LM adaptation framework, which augments PLMs with entities and their relations from the target domain, during fine-tuning without any further pre-training. To our knowledge, this is the first work that utilizes the structured knowledge for language model adaptation.
KALA: Knowledge-Augmented Language Model Adaptation
•  (2020) showed that adaptive pre-training of the PLM on the corpus of a target domain (Domain-adaptive Pre-training; DAPT) or a target task (Task-adaptive Pre-training; TAPT) improves its performance on domain-specific tasks. However, above approaches generally require a large amount of computational costs for pre-training.
KALA: Knowledge-Augmented Language Model Adaptation
Knowledge-aware LM Accompanied with increasing sources of knowledge (Vrandecic and Krötzsch, 2014), some prior works have proposed to integrate external knowledge into PLMs, to enhance their performance on tasks that require structured knowledge. For instance, ERNIE (Zhang et al., 2019) and KnowBERT (Peters et al., 2019) incorporate entities as additional inputs in the pretraining stage to obtain a knowledge-aware LM, wherein a pre-trained knowledge graph embedding from Wikidata (Vrandecic and Krötzsch, 2014) is used to represent entities. Entity-as-Experts (Févry et al., 2020) and LUKE (Yamada et al., 2020) use the entity memory that is pre-trained along with the LMs from scratch. ERICA (Qin et al., 2021) further uses the fact consisting of entities and their relations in the pre-training stage of LMs from scratch. Previous works aim to integrate external knowledge into the LMs during the pre-training step to obtain a universal knowledge-aware LM that requires additional parameters for millions of entities. In contrast to this, our framework aims to efficiently modify a general PLM for the domain-specific task with a linear modulation layer scheme discussed in Section 3.2, during fine-tuning.
KALA: Knowledge-Augmented Language Model Adaptation
Our goal is to solve Natural Language Understanding (NLU) tasks for a specific domain, with a knowledge-augmented Language Model (LM). We first introduce the NLU tasks we target, followed by the descriptions of the proposed knowledgeaugmented LM. After that, we formally define the ingredients for structured knowledge integration.
KALA: Knowledge-Augmented Language Model Adaptation
NLU tasks The goal of an NLU task is to predict the label y of the given input instance x, where the input x contains the sequence of tokens (Devlin et al., 2019)
KALA: Knowledge-Augmented Language Model Adaptation
, the objective is to maximize the log-likelihood as follows:
KALA: Knowledge-Augmented Language Model Adaptation
where f is an encoder of the PLM which outputs contextualized representation H from x, and g is a decoder which models the probability distribution p of the label y, with trainable parameters θ = (θ f , θ g ). To this end, we propose a function h(·; φ) which augments PLMs conditioned on the domain knowledge. Formally, the objective for a NLU task with our knowledge-augmented LM is given as follows:
KALA: Knowledge-Augmented Language Model Adaptation
where φ is parameters for the function h, E is the set of entities, M is the set of corresponding mentions, and G is a knowledge graph. In the following, we will describe the definition of the knowledgerelated inputs E, M, G, and the details of h(·, φ). We further construct the entity vocabulary E train = N i=1 E (i) , which consists of all entities appearing in the training dataset. However, at test time, we may encounter unseen entities that are not in E train . To tackle this, we regard unknown entities as the null entity e ∅ , so that ∀e ∈ E train ∪ {e ∅ }.
KALA: Knowledge-Augmented Language Model Adaptation
Definition 2 (Entity Memory). Given a set of all entities E train ∪ {e ∅ }, we represent them in the continuous vector (feature) space to learn meaningful entity embeddings. In order to implement this, we define the entity memory E ∈ R (|E train |+1)×d that comprises of an entity e ∈ R as a key and its embedding e ∈ R d as its value. Also, to access the value in the entity memory, we define the point-wise memory access function EntEmbed which takes an entity as an input. For instance, e = EntEmbed(New_York) returns the embedding of the New_York entity, and e = EntEmbed(e ∅ ) returns the zero embedding. This entity memory E is the part of the parameter φ used in function h.
KALA: Knowledge-Augmented Language Model Adaptation
Definition 3 (Knowledge Graph). Since the entity memory alone cannot represent relational information between entities, we further define a Knowledge Graph (KG) G that consists of a set of factual triplets {(h, r, t)}, where the head and the tail entities, h and t, are the elements of E, and a relation r is an element of a set of relations R: h, t ∈ E and r ∈ R. We assume that a preconstructed KG G (i) is given for each input x (i) , and provide the details of the KGs and how to construct them in Appendix A.
KALA: Knowledge-Augmented Language Model Adaptation
The remaining problem is how to augment a PLM by conditioning it on the domain-specific knowledge, through the function h. An effective approach to do so without stacking additional layers on top of the LM is to interleave the knowledge from h with the pre-trained parameters of the language model (Devlin et al., 2019) consisting of transformer layers (Vaswani et al., 2017). Before describing our interleaving method in detail, we first describe the Transformer architecture.
KALA: Knowledge-Augmented Language Model Adaptation
Transformer Given |x| token representations H l−1 = [h l−1 1 , . . . , h l−1 |x| ] ∈ R |x|×d from the layer l − 1 where d is the embedding size, each transformer block outputs the contextualized representations for all tokens. In detail, the l-th block consists of the multi-head self-attention (Attn) layer and the residual feed-forward (FF) layer as follows:
KALA: Knowledge-Augmented Language Model Adaptation
where LN is a layer normalization (Ba et al., 2016), σ is an activation function (Hendrycks and Gimpel, 2016), W 2 ∈ R d ×d and W 1 ∈ R d×d are weight matrices, and d is an intermediate hidden size. We omit the bias term for brevity.
KALA: Knowledge-Augmented Language Model Adaptation
Linear Modulation on Transformer An effective yet efficient way to fuse knowledge from different sources without modifying the original model architecture is to scale and shift the features of one source with respect to the data from another source (Dumoulin et al., 2018). This scheme of feature-wise affine transformation is effective on various tasks, such as language-conditioned image reasoning (Perez et al., 2018) 
KALA: Knowledge-Augmented Language Model Adaptation
Multi-Head Self-Attention Motivated by them, we propose to linearly transform the intermediate features after the layer normalization of the transformer-based PLM, conditioned on the knowledge sources E, M, G. We term this method as the Knowledge-conditioned Feature Modulation (KFM), described as follows:
KALA: Knowledge-Augmented Language Model Adaptation
where H l−1 ∈ R |x|×d is the matrix of hidden representations from the previous layer, • denotes the hadamard (element-wise) product, and Γ = [γ 1 , . . . , γ |x| ] ∈ R |x|×d , B = [β 1 , . . . , β |x| ] ∈ R |x|×d . Γ and B are learnable modulation parameters from the function h, which are conditioned by the entity representation. For instance, in Figure 3, γ and β for token 'New' are conditioned on the corresponding entity New_York. However, if tokens are not part of any entity (e.g., 'is'), γ and β for such tokens are fixed to 1 and 0, respectively. One notable advantage of our KFM is that multiple tokens associated to the identical entity are affected by the same modulation (e.g., 'New' and 'York' in Figure 3), which allows the PLM to know which adjacent tokens are in the same entity. This is important for representing the tokens of the domain entity (e.g., 'cod' and 'on'), since the original PLM might regard them as separate, unrelated tokens (See analysis in §5.5 with Figure 5). However, with our KFM, the PLM can identify associated tokens and embed them to be close to each other.
KALA: Knowledge-Augmented Language Model Adaptation
Then, how can we design such functional operations in h? The easiest way is to retrieve the entity embedding of e, associated to the typical to-ken, from the entity memory E, and then use the retrieved entity embedding as the input to obtain γ and β for every entity (See Figure 3). Formally, for each entity e ∈ E and its mention (m α , m ω ) ∈ M, v = EntEmbed(e)
KALA: Knowledge-Augmented Language Model Adaptation
(2)
KALA: Knowledge-Augmented Language Model Adaptation
where v is the retrieved entity embedding from the entity memory, h 1 , h 2 , h 3 , and h 4 are mutually independent Multi-Layer Perceptrons (MLPs) which return a zero vector 0 if e = e ∅ .
KALA: Knowledge-Augmented Language Model Adaptation
Although the simple access to the entity memory can retrieve the necessary entity embeddings for the modulation, this approach has obvious drawbacks as it not only fails to reflect the relations with other entities, but also regards unseen entities as the same null entity e ∅ . If so, all unseen entities are inevitably modulated by the same parameters even if they have essentially different meaning.
KALA: Knowledge-Augmented Language Model Adaptation
To tackle these limitations, we further consider the relational information between two entities that are linked with a particular relation. For example, the entity New_York alone will not give meaningful information. However, with two associated facts (New_York, instance of, city) and (New_York, country, USA), it is clear that New_York is a city in the USA. Motivated by this observation, we propose Relational Retrieval which leverages a KG G to retrieve entity embeddings from the memory, according to the relations defined in the given KG (See Figure 3, right).
KALA: Knowledge-Augmented Language Model Adaptation
More specifically, our goal is to effectively utilize the relations among entities in G, to improve the EntEmbed function in equation 2. We tackle this objective by utilizing a Graph Neural Network (GNN) which learns feature representations of each node using a neighborhood aggregation scheme (Hamilton et al., 2017), as follows:
KALA: Knowledge-Augmented Language Model Adaptation
where N (e; G) is a set of neighboring entities of the entity e, AGG is the function that aggregates embeddings of neighboring entities of e, and UPDATE is the function that updates the representation of e with the aggregated messages from AGG. However, simple aggregation (e.g., mean) cannot reflect the relative importance on neighboring nodes, thus we consider the attentive scheme (Velickovic et al., 2018;Brody et al., 2021) for neighborhood aggregation, to allocate weights to the target entity's neighbors by their importance. This scheme is helpful in filtering out less useful relations. Formally, we first define a scoring function ψ that calculates a score for every triplet (e i , r ij , e j ), which is then used to weigh each node during aggregation:
KALA: Knowledge-Augmented Language Model Adaptation
where σ is a nonlinear activation, e * ∈ R 4d is concatenated vector where denotes the concatenation, a ∈ R d and W ∈ R d×4d are learnable parameters, r ij ∈ R d is a embedding of the relation, and h e i ∈ R d is a context representation of the entity e i obtained from the intermediate hidden states of the LM 3 .
KALA: Knowledge-Augmented Language Model Adaptation
The scores obtained from ψ are normalized across all neighbors e j ∈ N (e i ; G) with softmax:
KALA: Knowledge-Augmented Language Model Adaptation
.
KALA: Knowledge-Augmented Language Model Adaptation
Then, we update the entity embedding with a weighted average of the neighboring nodes with α as an attention coefficient, denoted as follows:
KALA: Knowledge-Augmented Language Model Adaptation
We evaluate our model on two NLU tasks: Question Answering (QA) and Named Entity Recognition (NER). For QA, we use three domain-specific datasets: NewsQA (News, Trischler et al., 2017) and two subsets (Relation, Medication) of EMRQA (Clinical, Pampari et al., 2018). We use the Exact-Match (EM) and the F1 score as evaluation metrics. For NER, we use three datasets from different domains, namely CoNLL-2003 (News, Sang andMeulder, 2003), WNUT-17 (Social Media, Derczynski et al., 2017) and NCBI-Disease (Biomedical, Dogan et al., 2014). We use the F1 score as the evaluation metric. We report statistics and detailed descriptions of each dataset in Appendix B.2.
KALA: Knowledge-Augmented Language Model Adaptation
A direct baseline of our KALA is the adaptive pre-training, which is commonly used to adapt the PLM independent to the choice of a domain and task. Also, to compare ours against a more powerful baseline, we modify a recent method (Chen et al., 2020) that alleviates forgetting of PLM during fine-tuning. Details for each baseline we use are described as follows:
KALA: Knowledge-Augmented Language Model Adaptation
1. Vanilla Fine-Tuning (FT): A baseline that directly fine-tunes the LM on downstream tasks.
KALA: Knowledge-Augmented Language Model Adaptation
2. Fine-Tuning + more params: A baseline with one more transformer layer at the end of the  Table 2: Experimental results of the NER task on three different datasets with the BERT-base. The reported results are means and standard deviations over five different runs with an F1 score as a metric. The numbers in bold fonts denote the best score. † indicates the baseline under an extremely high computational resource setting (See Figure 1).
KALA: Knowledge-Augmented Language Model Adaptation
LM. We use this baseline to show that the performance gain of our model does not come from the use of additional parameters.
KALA: Knowledge-Augmented Language Model Adaptation
3. Task 7. KALA (relational): Our full model that uses KGs to perform relational retrieval from the entity memory.
KALA: Knowledge-Augmented Language Model Adaptation
We use the uncased BERT-base (Devlin et al., 2019) as the base PLM for all our experiments on QA and NER tasks. For more details on training and implementation, please see the Appendix B.
KALA: Knowledge-Augmented Language Model Adaptation
Performance on QA and NER tasks On both extractive QA and NER tasks, our KALA outperforms all baselines, including TAPT and TAPT+RedcAdam (Gururangan et al., 2020;Chen et al., 2020), as shown in Table 1 and 2. These results show that our KALA is highly effective for the language model adaptation task. KALA also largely outperforms DAPT (Gururangan et al., 2020) which is trained with extra data and requires a significantly higher computational cost compare to KALA (See Figure 1 for the plot of efficiency, discussed in Section 5.3).
KALA: Knowledge-Augmented Language Model Adaptation
Effect of Using more Parameters One may suspect whether the performance of our KALA comes from the increment of parameters. However, the experimental results in Table 1 and 2 show that increasing the parameters for PLM during fine-tuning (+ more params) yields marginal performance improvements over naive fine-tuning. This result confirms that the performance improvement of KALA is not due to the increased number of parameters.
KALA: Knowledge-Augmented Language Model Adaptation
The performance gap between KALA (relational) and KALA (point-wise) shows the effectiveness of relational retrieval for language model adaptation, which allows us to incorporate relational knowledge into the PLM. The relational retrieval also helps address unseen entities, as discussed in Section 5.4.
KALA: Knowledge-Augmented Language Model Adaptation
We perform an ablation study to see how much each component contributes to the performance gain.
KALA: Knowledge-Augmented Language Model Adaptation
We first analyze the effect of feature modulation parameters (i.e., gamma and beta) in transformers by ablating a subset of them in Table 3, in which we observe that using both     (Qin et al., 2021). Note that, most of them were proposed for improving pre-training from scratch, while we adapt them for fine-tuning under our KALA framework (The details are given in Appendix B.4). As shown in Table 4, our KFM used in KALA outperforms all variants, demonstrating the effectiveness of feature modulation in the middle of transformer layers for fine-tuning.
KALA: Knowledge-Augmented Language Model Adaptation
Although we believe our experimental results on more params on NewsQA). Thus, we believe that our KALA would be useful to any PLMs, not depending on specific PLMs. Figure 1 illustrates the performance and training FLOPs of KALA against baselines on the NewsQA dataset. We observe that the performance of TAPT decreases with the increased number of iterations, which could be due to forgetting of the knowledge from the PLM. On the other hand, DAPT, while not suffering from performance loss, requires huge computational costs as it trains on 112 times larger data for further pre-training (See Appendix B.3 for detailed explanations on training data). On the other hand, our KALA outperforms DAPT without using external data, while requiring 17 times fewer computational costs, which shows that KALA is not only effective but also highly efficient.
KALA: Knowledge-Augmented Language Model Adaptation
To further compare the efficiency in various aspects, we report GPU memory, training wall time, and training FLOPs for baselines and ours in Table 6. Through this, we verify that our KALA is more efficient to train for language model adaptation settings than baselines. Note that the resource requirement of KALA could be further reduced by adjusting the size of the entity memory (e.g., removing less frequent entities). Therefore, to show the flexibility of our KALA on the typical resource constraint, we provide the experimental results on two different settings (i.e., tuning the number of entities in the entity memory) -KALA with memory size of 200 and 62.8k (full memory) in Appendix C.6.   
KALA: Knowledge-Augmented Language Model Adaptation
One remarkable advantage of our KALA is its ability to represent an unseen entity by aggregating features of its neighbors from a given KG. To analyze this, we first divide all contexts into one of Seen and Unseen, where Seen denotes the context with less than 3 unseen entities, and then measure the performance on the two subsets. As shown in Figure 4, we observe that the performance gain of KALA over the baselines is much larger on the Unseen subset, which demonstrates the effectiveness of KALA's relational retrieval scheme to represent unseen entities. DAPT also largely outperforms fine-tuning and TAPT as it is trained on an extremely large external corpus for adaptive pre-training. However, KALA even outperforms DAPT in most cases, verifying that our knowledgeaugmentation method is more effective for tackling domain-specific tasks. The visualization of embeddings of seen and unseen entities in Figure 2 shows that KALA embeds the unseen entities more closely to the seen entities 4 , which explains KALA's good performance on the Unseen subset.
KALA: Knowledge-Augmented Language Model Adaptation
To better see how our KFM ( §3.2) works, we show the context and its fact, and then visualize representations from the PLM modulated by the KFM. 4 We quantitatively measure the mean of cosine distance of each unseen entity to its nearest seen entity, observing that KALA embeds unseen 1.5 times more closer to seen than DAPT (i.e., 0.07 for KALA vs 0.11 for DAPT for distance).
KALA: Knowledge-Augmented Language Model Adaptation
As shown in Figure 5 right, the token '##on' is not aligned with their corresponding tokens, such as 'ex' (for exon) and 'cod' (for codon), in the baseline. However, with our feature modulation that transforms multiple tokens associated with the single entity equally, the two tokens (e.g., ('ex', '##on')), composing one entity, are closely embedded. Also, while the baseline cannot handle the unseen entity consisting of three tokens: 're', '##tina', and '##l', KALA embeds them closely by representing the unseen retinal from the representation of its neighborhood gene derived by the domain knowledge -(retinal, instance of, gene).
KALA: Knowledge-Augmented Language Model Adaptation
Our KALA framework is also applicable to encoder-decoder PLMs by applying the KFM to the encoder. Therefore, we further validate KALA's effectiveness on the encoder-decoder PLMs on the generative QA task (Lee et al., 2021) with T5small (Raffel et al., 2020). Table 7 shows that KALA largely outperforms baselines even with such a generative PLM.
KALA: Knowledge-Augmented Language Model Adaptation
In this paper, we introduced KALA, a novel framework for language model adaptation, which modulates the intermediate representations of a PLM by conditioning it with the entity memory and the relational facts from KGs. We validated KALA on various domains of QA and NER tasks, on which KALA significantly outperforms relevant baselines while being computationally efficient. We demonstrate that the success of KALA comes from both KFM and relational retrieval, allowing the PLM to recognize entities but also handle unseen ones that might frequently appear in domain-specific tasks. There are many other avenues for future work, including the application of KALA on pre-training of knowledge-augmented PLMs from scratch.
KALA: Knowledge-Augmented Language Model Adaptation
Enhancing the domain converge of pre-traind language models (PLMs) with external knowledge is increasingly important, since the PLMs cannot observe all the data during training and cannot memorize all the necessary knowledge for solving down-stream tasks. Our KALA contributes to this problem by augmenting domain knowledge graphs for PLMs. However, we have to still consider the accurateness of knowledge, i.e., the fact in the knowledge graph may not be correct, which affects the model to generate incorrect answers. Also, the model's prediction performance is still far from optimal. Thus, we should be aware of model's failure from errors in knowledge and prediction, especially on high-risk domains (e.g., biomedicine).  
KALA: Knowledge-Augmented Language Model Adaptation
"text": "Arvane Rezai", "start": 30, "end": 43, "id": 228998 "h": 11578, "r": "P3373", "t": 228998 Figure 6: Visual diagram of the KG construction pipeline used in this work. The entity format is composed of its corresponding text in the data, its character-level mention boundary, and its wikidata id. The fact format is composed of the head, relation, and tail, where head and tail entities are represented with their wikidata ids following the entity format.  
KALA: Knowledge-Augmented Language Model Adaptation
In this work, we propose to use the Knowledge Graph (KG) that can define the relational information among entities that only appear in each dataset. However, unfortunately, most of the task datasets do not contain such relational facts on its context, thus we need to construct them manually to obtain the knowledge graph. In this section, we explain the way of constructing the knowledge graph that we used, consisting of facts of entities for each context in the task dataset. Relation extraction is the way how we obtain the factual knowledge from the text of the target dataset. To do so, we first need to extract entities and their corresponding mentions from the text, and then link it to the existing entities in wikidata (Vrandecic and Krötzsch, 2014). In order to do this, we use the existing library named as spaCy 5 , and opensourced implementation of Entity Linker 6 . To sum up, in our work, a set of entities E (i) and corresponding mentions M (i) for the given input x (i) are obtained through this step. Regarding a concrete example, please see format (a) in Figure 6. In the example, "Text" indicates the entity mention within the input x, the "start" and "end" indicates its mention position denoted as (m α , m ω ), and "id" indicates the wikidata id for the entity identification used in the next step.
KALA: Knowledge-Augmented Language Model Adaptation
To extract the relation among entities that we obtained above, we use the scheme of Relation Extraction (RE). In other words, we use the trained 5 https://spacy.io/ 6 https://github.com/egerber/spaCy-entity-linker RE model to build our own knowledge base (KB) instead of using the existing KG directly from the existing general-domain KB 7 . Specifically, we first fine-tune the BERT-base model (Devlin et al., 2019) for 2 epochs with 600k distantly supervised data used in Qin et al. (2021), where the Wikipedia document and the Wikidata triplets are aligned. Then, we use the fine-tuned BERT model to extract the relations between entity pairs in the text. We use the model with a simple bilinear layer on top of it, which is widely used scheme in the relation extraction literature (Yao et al., 2019). For an example of the extracted fact, please see format (b) in Figure 6. In the example, "h" denotes the wikidata id of the head entity, "r" denotes the wikidata id of the extracted relation, and "t" denotes the wikidata id of the tail entity. In the relation extraction, the model returns the categorical distribution over the top 100 frequent relations. In general, the relation of top-1 probability is used as the relation for the corresponding entity pair. However, this approach sometimes results in predicting no_relation on most entity pairs. Thus, to obtain more relations, we further use the relation of top-2 probability in the case where no_relation has a top-1  Table 9: QA dataset statistics. We report the number of contexts and questions (i.e., # Context and # Question), with the average length of contexts (i.e., C. Length) where the length is measured as the number of tokens after wordpiece tokenization.
KALA: Knowledge-Augmented Language Model Adaptation
probability but the top-2 probability is larger than a certain threshold (e.g., > 0.1). In Figure 6, we summarize our KG construction pipeline. In Table 8, we report the hyperparameters related to our KG construction.
KALA: Knowledge-Augmented Language Model Adaptation
In this section, we introduce the detailed setups for our models and baselines used in Table 1, 2, and 4.
KALA: Knowledge-Augmented Language Model Adaptation
We use the Pytorch (Paszke et al., 2019) for the implementation of all models. Also, to easily implement the language model, we use the huggingface library (Wolf et al., 2020) containing various transformer-based pre-trained language models (PLMs) and their checkpoints.
KALA: Knowledge-Augmented Language Model Adaptation
Details for KALA In this paragraph, we describe the implementation details of the components, such as four linear layers in the proposed KFM, architectural specifications in the attentionbased GNN, and initialization of both the entity memory and relational embeddings, in the following. In terms of the functions h 1 , h 2 , h 3 , and h 4 in the KFM of Equation 2, we use two linear layers with the ReLU (Nair and Hinton, 2010) activation function, where the dimension is set to 768.
KALA: Knowledge-Augmented Language Model Adaptation
For relational retrieval, we implement the novel GNN model based on GATv2 (Brody et al., 2021) provided by the torch-geometric package (Fey and Lenssen, 2019). Specifically, we stack two GNN layers with the RELU activation function and also use the dropout with a probability of 0.1. For attention in our GNN, we mask the nodes of the null entity, so that the attention score becomes zero for them. Moreover, to obtain the context representation of the entity (See Footnote 3 in the main paper) used in the GNN attention, we use the scatter operation 8 for reduced computational cost.
KALA: Knowledge-Augmented Language Model Adaptation
For Entity Memory, we experimentally found that initializing the embeddings of the entity memory with the contextualized features obtained from 8 https://github.com/rusty1s/pytorch_scatter the pre-trained language model could be helpful. Therefore, the dimension of the entity embedding is set to the same as the language model d = 768. For relation embeddings, we randomly initialize them, where the dimension size is set to 128.
KALA: Knowledge-Augmented Language Model Adaptation
Location of KLM in the PLM Note that, the number and location of the KFM layers inside the PLM are hyperparameters. However, we empirically found that inserting one to three KFM layers at the end of the PLM (i.e., after the 9th -11th layers of the BERT-base language model) is beneficial to the performance (See Appendix C.4 for experiments on diverse layer locations).
KALA: Knowledge-Augmented Language Model Adaptation
Here we describe the dataset details with its statistics for two different tasks: extractive question answering (QA) and named entity recognition (NER).
KALA: Knowledge-Augmented Language Model Adaptation
Question Answering We evaluate models on three domain-specific datasets: NewsQA, Relation, and Medication. Notably, NewsQA (Trischler et al., 2017) is curated from CNN news articles. Relation and Medication are originally part of the emrQA (Pampari et al., 2018), which is an automatically constructed question answering dataset based on the electrical medical record from n2c2 challenges 9 . However, Yue et al. (2020) extract two major subsets by dividing the entire dataset into Relation and Medication and suggest the usage of sampled questions from the original em-rQA dataset. Following the suggestion of Yue et al.
KALA: Knowledge-Augmented Language Model Adaptation
(2020), we use only 1% of generated questions of Relation for training, validation, and testing. Also, we only use 1% of generated questions of Medication for training and use 5% of generated questions of Medication for validation and testing. Since the original emrQA is automatically generated based on templates, the quality is poor -it means that the original emrQA dataset was inappropriate to evaluate the ability of the model to reason over the clinical text since the most of questions can be    bels. The NCBI-Disease dataset consists of the 793 PubMed articles from the biomedical domain, which contains 6,892 disease mentions and 790 disease concepts, and also has 3 class labels. The statistics of training, validation, and test sets are provided in Table 11.
KALA: Knowledge-Augmented Language Model Adaptation
All experiments are constrained to be done with a single 12GB Geforce RTX 2080 Ti GPU for fairness in terms of memory and the availability on the academic budget, except for the DAPT and generative QA which use a single 48GB Quadro 8000 GPU. KALA training needs 3 hours in wall time with a single GPU. For all experiments, we select the best checkpoint on the validation set. For the summary of training setups, please see Table 10 and 12.
KALA: Knowledge-Augmented Language Model Adaptation
Fine-tuning Setup In the following three paragraphs, we explain the setting of fine-tuning for QA, NER, and generative QA tasks. For all experiments on extractive QA tasks, we fine-tune the Pre-trained Language Model (PLM) for 2 epochs with the weight decay of 0.01, learning rate of 3e-5, maximum sequence length of 384, batch size of 12, linear learning rate decay of 0.06 warmup rate, and half precision (Micikevicius et al., 2018).
KALA: Knowledge-Augmented Language Model Adaptation
For all experiments on NER tasks, we finetune the PLM for 20 epochs, where the learning rate is set to 5e-5, maximum sequence length is set to 128, and batch size is set to 32. We use AdamW (Loshchilov and Hutter, 2019) as an optimizer using BERT-base as the PLM.
KALA: Knowledge-Augmented Language Model Adaptation
For the generative QA task in Table 7, we finetune the T5-small (Raffel et al., 2020) for 4 epochs with the learning rate of 1e-4, maximum sequence length of 512, and batch size of 64. We also use the Adafactor (Shazeer and Stern, 2018) optimizer. Instead of training with the same optimizer as in BERT for QA and NER, we instead use the independent AdamW optimizer with the learning rate of 1e-4 and weight decay of 0.01 to train the KALA module with T5.
KALA: Knowledge-Augmented Language Model Adaptation
Adaptive Pre-training Setup In this paragraph, we describe the experimental settings of adaptive pre-training baselines, namely TAPT, TAPT (+ RecAdam), and DAPT. For QA tasks, we further pre-train the PLM for {1,3,5,10} epochs and then report the best performance among them. Specifically, reported TAPT result on NewsQA, Relation, and Medication are obtained by 1 epoch of further pre-training. We use the weight decay of 0.01, learning rate of 5e-5, maximum sequence length of 384, batch size of 12, and linear learning rate decay of 0.06 warmup rate, with a half-precision. Also, the masking ratio for the pre-training objective is set to 0.15, following the existing strategy introduced in the original BERT paper (Devlin et al., 2019).
KALA: Knowledge-Augmented Language Model Adaptation
For NER tasks, we further pre-train the PLM for 3 epochs across all datasets. In particular, the learning rate is set to 5e-5, batch size is set to 32, and the maximum sequence length is set to 128. We also use AdamW (Loshchilov and Hutter, 2019) as the optimizer for all experiments.
KALA: Knowledge-Augmented Language Model Adaptation
In the case of T5-small for generative QA in Table 7, we further pre-train the PLM for 4 epochs with the learning rate of 0.001, batch size of 64, maximum sequence length of 384, and Adafac-tor (Shazeer and Stern, 2018) optimizer.
KALA: Knowledge-Augmented Language Model Adaptation
Regarding the setting of TAPT (+ RecAdam) on all tasks, we follow the best setting in the original paper (Chen et al., 2020) -sigmoid as an annealing function with annealing parameters: k = 0.5, t 0 = 250, and the pretraining coefficient of 5000.
KALA: Knowledge-Augmented Language Model Adaptation
For training with DAPT, we need an external corpus having a large amount of data for adaptive pre-training. Thus, we first choose the datasets of two domains -News and Medical. Specifically, as the source of corpus for the News domain, we use the sampled set of 10 million News from the RealNews dataset used in Gururangan et al. (2021). As the source of corpus for the Medical domain, we use the set of approximately 100k passages from the Medical textbook provided in Jin et al. (2020). The size of pre-training data used in DAPT is much larger than TAPT. In other words, for experiments on NewsQA, TAPT only uses fine-tuning contexts containing 5.8 million words from the NewsQA training dataset, while DAPT uses more than a hundred times larger data -enormous contexts containing about 618 million words from the RealNews database. For both News and Medical domains, we further pre-train the BERT-base model for 50 epochs with the batch size of 64, to match the similar computational cost used in Gururangan et al. (2020). Other experimental details are the same as TAPT described above.
KALA: Knowledge-Augmented Language Model Adaptation
In this subsection, we describe the details of architectural variants reported in Section 5.1. For all variants, we use the same KGs used in KALA.
KALA: Knowledge-Augmented Language Model Adaptation
Entity-as-Experts (Févry et al. (2020); EaE) utilizes the entity memory similar to our work, but they use the parametric dense retrieval more like the memory neural network (Sukhbaatar et al., 2015). Similar to Févry et al. (2020); Verga et al. (2021), we change the formulation of query and memory retrieval by using the mention representation of the entity from the intermediate hidden states of PLMs, which is formally defined as follows:
KALA: Knowledge-Augmented Language Model Adaptation
where h e represents the average of token representations of the entity mention m = (m α , m ω ). We also give the supervised retrieval loss (ELLoss in Févry et al. (2020)), when training the EaE model. With this retrieval, EaE also can represent the unseen entity e / ∈ E train if we know the mention boundary of the given entity on the context. We believe it is expected to work well, if the entity memory is pre-trained on the enormous text along with the pre-training of the language model from the scratch. However, it might underperform for the language model adaptation scenario, since it can fall into the problem of circular reasoning -the PLM does not properly represent the unseen entity, but it should predict which entity it is similar from the representation. Regarding the integration of the knowledge from the entity memory into the PLM, the retrieved entity representation v is simply added (Peters et al., 2019) to the hidden representations H after the transformer block as follows:
KALA: Knowledge-Augmented Language Model Adaptation
where h is Multi-Layer Perceptrons (MLPs).
KALA: Knowledge-Augmented Language Model Adaptation
Adapter (Houlsby et al., 2019) is introduced to fine-tune the PLM only with a few trainable parameters, instead of fine-tuning the whole parameters of the PLM. To adapt this original implementation into our KALA framework, we replace our Knowledge-conditioned Feature Modulation with it, where the Adapter is used as the knowledge integration module. We interleave the layer of Adapter after the feed-forward layer (F F ) and before the residual connection of the transformer block. Also, instead of only providing the LM hidden states as an input, we concatenate the knowledge representation in Equation 3 to the LM hidden states. Note that we fine-tune the whole parameters following our KALA setting, unlike fine-tuning the parameters of only Adapter layers in Houlsby et al. (2019).
KALA: Knowledge-Augmented Language Model Adaptation
ERNIE (Zhang et al., 2019) is a notable PLM model that utilizes the external KB as an input for the language model. The key feature of ERNIE can be summarized into two folds. First, they use the multi-head self-attention scheme (Vaswani et al., 2017) to contextualize the input entities. Second, ERNIE fuses the entity representation at the end of the PLM by adding it to the corresponding language representation. We assume that those two features are important points of ERNIE. Therefore, instead of using a Graph Neural Network (GNN) layer, we use a multi-head self-attention layer to contextualize the entity embeddings. Then, we add it to a representation of the entity from the PLM, which is the same as the design in equation 5.  uses knowledge as an external input in the fine-tuning stage for extractive QA. Since they have a typical layer for integrating existing KB (Miller, 1995;Carlson et al., 2010) with the PLM, we only adopt the self-matching layer as the architecture variant of the KFM layer used in our KALA framework. The computation of the self-matching matrix in KT-Net is costly, i.e., it requires a large computational cost that is approximately 12 times larger than KALA.
KALA: Knowledge-Augmented Language Model Adaptation
ERICA (Qin et al., 2021) uses contrastive learning in LM pre-training to reflect the relational knowledge into the language model. We use the Entity Discrimination task from ERICA on the primary task of fine-tuning. We would like to note that, as reported in Section 5 of the original paper (Qin et al., 2021), the use of ERICA on fine-tuning has no effect, since the size and diversity of entities and relations in downstream training data are limited. Such limited information rather harms the performance, as it can hinder the generalization. In other words, contrastive learning cannot reflect the entity and relation in the test dataset.
KALA: Knowledge-Augmented Language Model Adaptation
In this subsection, we give detailed descriptions of how the FLOPs in Figure 1 are measured. We majorly follow the script from the ELECTRA (Clark et al., 2020) repository to compute the approximated FLOPs for all models including ours. For FLOPs computation of our KALA, we additionally include the FLOPs of the entity embedding layer, linear layers for h 1 , h 2 , h 3 , h 4 , and GNN layer. Since the GNN layer is implemented based on the sparse implementation, we first calculate the FLOPs of the message propagation over one edge, and then multiply it to the average number of edges per node. Also, in terms of the computation on mentions, we consider the maximum sequence length of the context rather than the average number of mentions, to set the upper bound of FLOPs for our KALA. Note that, in NewsQA training data, the average number of nodes is 57, the average number of edges for each node is 0.64, and the average number of mentions in the context is 92.68.
KALA: Knowledge-Augmented Language Model Adaptation
In this section, we provide the analyses on the forgetting of TAPT, entity memory, number of entities and facts, location of the KLM layer, and values of Gamma and Beta. 
KALA: Knowledge-Augmented Language Model Adaptation
In Figure 1, we observe that the performance of TAPT decreases as the number of training steps increases. To get a concrete intuition on this particular phenomenon, we analysis what happens in the Pre-trained Language Model (PLM), when we further pre-train it on the task-specific corpus. Specifically, in Figure 7, we visualize the Masked Language Model (MLM) loss of TAPT on both domain-specific corpus from the Relation dataset and general corpus from the sampled Wikipedia documents during the adaptive pre-traing. As Figure 7 shows, the test MLM loss increases while the training MLM loss persistently increases as the training step increases. This result indicates that TAPT on domain-specific corpus may yield the catastrophic forgetting of the general knowledge in the PLM.
KALA: Knowledge-Augmented Language Model Adaptation
In this subsection, we analyze how the size of entity memory affects the performance of our KALA. In Figure 8, we plot the performance of KALA on the NewsQA dataset by varying the number of entity elements in the memory. Note that, we reduce the size of the entity memory by eliminating the entity appearing fewer times. Thus, the results are obtained by only considering the entities that appear more than [1000, 100, 10, 5, 0] times, e.g., 0 means the model with full entity memory. As shown in Figure 8, we observe that the size of the entity memory is larger, the performance of our KALA is better in general. However, interestingly, we also observe that the smallest size of the entity memory shows decent performance, which might be due to the fact that some parameters in the entity memory are stale. For more discussions on it including visualization, please refer to Appendix D.2. Finally, we would like to note that, in Figure 1, we report the performance of our KALA in the case of [1000, 5, 0] (i.e., considering entities appearing more than [1000, 5, 0] times).
KALA: Knowledge-Augmented Language Model Adaptation
In this subsection, we aim to analyze which numbers of entities and facts per context are appropriate to achieve good performance in NER tasks. Specifically, we first collect the contexts having more than or equal to the k number of entities (or facts), and then calculate the performance difference from our KALA to the fine-tuning baseline. As shown in Figure 9, while there are no obvious patterns, performance improvements from the baseline are consistent across a varying number of entities and facts. This result suggests that our KALA is indeed beneficial when entities and facts are given to the model, whereas the appropriate number of entities and facts to obtain the best performance against the baseline is different across datasets. 
KALA: Knowledge-Augmented Language Model Adaptation
In the main paper and Appendix B.1, we describe that the location of the KFM layer inside the PLM architecture is the hyperparameter. However, someone might wonder which location of KFM yields the best performance, and what is the reason for this. Therefore, in this section, we analyze where we obtain the best performance in various locations of the KFM layer on the NewsQA dataset. Specifically, in Figure 10, we show the performance of our KALA with varying the location of the KFM layer insider the BERT-base model. The results demonstrate that the model with the KFM on the last layer of the BERT-base outperforms all the other choices. This might be because, as the final layer of the PLM is generally considered as the most task-specific layer, our KFM interleaved in the latest layer of BERT expressively injects the task-specific information from the entity memory and KGs, to such a task-specific layer.
KALA: Knowledge-Augmented Language Model Adaptation
To see how much amount of value on gamma and beta is used to shift and scale the intermediate hidden representations in transformer layers, we visualize the modulation values, namely gamma and beta, in Figure 11. We first observe that, as shown in Figure 11, the distribution of values of gamma and beta approximately follow the Gaussian dis-tribution, with zero mean for beta and one mean for gamma. Also, we notice that the scale of values remain nearly around the mean point, which suggests that the small amount of shifting to intermediate hidden representations on transformer layers is enough to contribute to the performance gain, as we can see in the main results of Table 1, 2.
KALA: Knowledge-Augmented Language Model Adaptation
While we provide the efficiency on FLOPs in Figure 1, we further provide the efficiency on GPU memory, wall time, and FLOPs for training each method in Table 6. Specifically, we measure the computational cost on the NewsQA dataset with BERT-base, where we use the single Geforce RTX 2080 Ti GPU on the same machine. For our KALA, as we can flexibly manage the cost of GPU memory by reducing the number of entities in entity memory (See Figure 8 with Appendix C.2 for more analysis on the effects of the size of entity memory), we provide the experimental results on two settings -KALA with memory size 0.2k and 62.8k (full memory). As shown in Table 6, we confirm that the computational cost of our KALA with the full memory is similar to the cost of the more params baseline that uses one additional transformer layer on top of BERT-base. However, by reducing the number of entities in the memory, we can achieve better efficiency than more params in terms of GPU memory and FLOPs. Also, we observe that the training cost (i.e., Wall Time and FLOPs) of TAPT and DAPT is high, especially on DAPT, thus we verify that our KALA is more efficient to train for domain adaptation settings.
KALA: Knowledge-Augmented Language Model Adaptation
Here we provide the frequency distribution of entities, additional case studies, and more illustrations of textual examples and embedding spaces. space in Figure 2 right, we further visualize them including the missing baselines of Figure 2, such as Fine-tuning or TAPT, in Figure 12 on the NCBI-Disease dataset. Similar to Figure 2, we observe that all baselines fail to closely embed the unseen entities in the representation space of seen entities. While this visualization result does not give a strong evidence of why our KALA outperforms other baselines, we clearly observe that KALA is beneficial to represent unseen entities in the feature space of seen entities, which suggests that such an advantage of our KALA helps the PLM to generalize over the test dataset, where the context contains unseen entities.
KALA: Knowledge-Augmented Language Model Adaptation
We visualize the frequency of entities in Figure 13 and 14. The entity frequency denotes the number of mentions of their associated entities within the entire text corpus of the training dataset. As shown in Figure 13 and 14 of QA and NER datasets, the entity frequency follows the long-tail distribution, where most entities appear a few times. For instance, in the NewsQA dataset, more than 20k entities among entire 60k entities appear only once in the training dataset, whereas one entity (CNN 10 ) appears approximately 20k times. This observation suggests that most of the elements in the entity memory are not utilized frequently. In other words, only few entities are accurately trained with many training instances, whereas there exists the stale embeddings which are rarely updated. This observation raises an interesting research question on the efficient usage of the entity memory, as we can see in Figure 8 that the small size of entity memory could result in the better performance (See Appendix C.2). We leave the more in-depth analysis on the entity memory as the future work. 
KALA: Knowledge-Augmented Language Model Adaptation
In addition to the case study in Figure 5, we further show the case on the question answering task in Figure 15, like in Section 5.5, With this example, we explain how the factual knowledge in KGs could be utilized to solve the task via our KALA. The question in the example is "who was kidnapped because of her neighbor". We observe that DAPT answers this question as Araceli Valencia. This prediction may come from matching the word 'her' in the question to the feminine name 'Araceli Valencia' in the context. In contrast, our KALA predicts the Jaime Andrade as an answer, which is the ground truth. We suspect that this might be because of the fact "(Jaime Andrade, spouse, Valencia)" in the knowledge graph, which relates the 'Valencia' to the 'Jaime Andrade'. Although it is not clear how it directly affects the model's performance, we can reason that KALA can successfully answer the question by utilizing the existing facts.
KALA: Knowledge-Augmented Language Model Adaptation
In Figure 16 and 17, we visualize the examples of the context with its seen and unseen entities and its relational facts. We first confirm that the quality of facts is moderate to use. For instance, in the first example of Figure 16, the fact in the context that Omar_bin_Laden is son of Osama_bin_Laden, is also appeared in the knowledge graph. In addition, we observe that there are facts that link unseen entities to the seen entities in both Figure 16 and 17. Thus, while some of the facts in the knowledge graph are not accurate, we can represent the unseen entities with their relation to the seen entities. We expect that there is a still room to improve in terms of the quality of KGs, allowing our KALA to modulate the entity representation more accurately. We leave the study on this as the future work.
KALA: Knowledge-Augmented Language Model Adaptation
Context PHOENIX, Arizona (CNN) -Jamie Andrade had just gotten out of the shower when the men came to snatch him. Jamie Andrade was kept in this closet for three days without food or water, police say. His wife, Araceli Valencia, was mopping the kitchen in … (ellipse)…
KALA: Knowledge-Augmented Language Model Adaptation
Question who was kidnapped because of her neighbour? Answer Jaime Andrade 
KALA: Knowledge-Augmented Language Model Adaptation
DAPT prediction Araceli Valencia, KALA prediction Jaime Andrade Figure 15: A textual example from NewsQA with predictions from each method (DAPT and KALA), and also the T-SNE plot of contextualized representations from the last layer of BERT obtained by each method. Grey dots indicate tokens without any mentions, and dots in other colors indicate tokens with mentions to the entity. We also represent sampled facts in Knowledge Graph we used. Blue text indicates the mention of seen entities and red text indicates the mention of unseen entities. The fact is represented as the format of (head, relation, tail). Text with yellow background indicates the ground truth answer span.
KALA: Knowledge-Augmented Language Model Adaptation
Context MADRID, Spain (CNN) -One of Osama bin Laden's sons has been denied asylum in Spain, an Interior Ministry spokeswoman told CNN on Wednesday. Omar bin Laden pictured earlier this year during television interview in Rome, Italy. Omar bin Laden, who is in his late 20s, stepped off a plane at Madrid's Barajas International Airport during a stopover late Monday and informed authorities that he planned to request political asylum, the spokeswoman said. Bin Laden has publicly called on his father to abandon terrorism. He prepared his formal asylum request Tuesday at the airport with the help of a translator, filing it around 1 p.m., the spokeswoman said. The Interior Ministry, which had 72 hours to reply to the request, was required to seek the opinion of the U.N. High Commissioner for Refugees on the matter. The UNHCR recommended … (ellipse) … Williams, who returns to No.2 in the world behind younger sister Serena on Monday, lost the opening set in just 27 minutes and then failed to take advantage of a 4-1 lead in the. "I just cannot believe this," world number 24 Rezai -who must now enter calculations for the French Open -told reporters. "Venus played very well and I've always respected her as a player and a champion. I just tried my best today and it worked well for me." Williams, who was looking to secure her 44 th career title, only converted two of her 13 break points in the batch -a statistic that contributed greatly to her defeat.  
KALA: Knowledge-Augmented Language Model Adaptation
The adenomatous polyposis coli ( APC ) tumour -suppressor protein controls the Wnt signalling pathway by forming a complex with glycogen synthase kinase 3beta ( GSK -3beta ), axin / conductin and betacatenin.
KALA: Knowledge-Augmented Language Model Adaptation
(complex, subclass of, protein) (GSK, instance of, protein) (glycogen, instance of, protein) (APC, instance of, protein)
KALA: Knowledge-Augmented Language Model Adaptation
Context HLA typing for HLA -B27, HLA -B60, and HLA -DR1 was performed by polymerase chain reaction with sequence -specific primers, and zygosity was assessed using microsatellite markers.
KALA: Knowledge-Augmented Language Model Adaptation
(microsatellite, subclass of, primers) (DR1, instance of, microsatellite) (microsatellite, subclass of, typing)
KALA: Knowledge-Augmented Language Model Adaptation
Context We identified four germline mutations in three breast cancer families and in one breast -ovarian cancer family. among these were one frameshift mutation, one nonsense mutation, one novel splice site mutation, and one missense mutation.
KALA: Knowledge-Augmented Language Model Adaptation
(frameshift mutation, subclass of, Germline mutations) (Nonsense mutation, subclass of, Germline mutations) (splice site mutation, subclass of, Germline mutations) (missense mutations, subclass of, Germline mutations) (Nonsense mutation, subclass of, cancers) (frameshift mutation, subclass of, cancers) (missense mutations, subclass of, cancers)
KALA: Knowledge-Augmented Language Model Adaptation
Context A nonsense mutation in exon 17 ( codon 556 ) of the RB1 gene was found to be present homozygously in both the retinal and the pineal tumours.
KALA: Knowledge-Augmented Language Model Adaptation
(retinal, instance of, gene) (Nonsense mutation, subclass of, gene)
KALA: Knowledge-Augmented Language Model Adaptation
Context Sixteen different p16 germline mutations were found in 21 families, while one germline mutation, Arg24His, was detected in the CDK4 gene.
KALA: Knowledge-Augmented Language Model Adaptation
(p16, subclass of, Germline mutations) (Germline mutations, subclass of, gene) (p16, instance of, gene)
KALA: Knowledge-Augmented Language Model Adaptation
Context Aspartylglucosaminuria ( AGU ) is a rare disorder of glycoprotein metabolism caused by the deficiency of the lysosomal enzyme aspartylglucosaminidase ( AGA ).
KALA: Knowledge-Augmented Language Model Adaptation
(Aspartylglucosaminuria, subclass of, deficiency)
KALA: Knowledge-Augmented Language Model Adaptation
Context Detection of heterozygous carriers of the ataxiatelangiectasia ( ATM ) gene by G2 phase chromosomal radiosensitivity of peripheral blood lymphocytes.
KALA: Knowledge-Augmented Language Model Adaptation
(ATM, instance of, gene) (G2 phase, part of, blood) (G2 phase, instance of, gene)
KALA: Knowledge-Augmented Language Model Adaptation
Context Recently, we reported five Austrian families with generalized atrophic benign epidermolysis bullosa who share the same COL17A1 mutation. 