Constructing Datasets
for Multi-hop Reading Comprehension Across Documents
Johannes Welbl

Pontus Stenetorp

Sebastian Riedel

arXiv:1710.06481v1 [cs.CL] 17 Oct 2017

University College London
{j.welbl,p.stenetorp,s.riedel}@cs.ucl.ac.uk

Abstract
Most Reading Comprehension methods limit
themselves to queries which can be answered
using a single sentence, paragraph, or document. Enabling models to combine disjoint
pieces of textual evidence would extend the
scope of machine comprehension methods,
but currently there exist no resources to train
and test this capability. We propose a novel
task to encourage the development of models
for text understanding across multiple documents and to investigate the limits of existing
methods. In our task, a model learns to seek
and combine evidence – effectively performing multi-hop (alias multi-step) inference. We
devise a methodology to produce datasets for
this task, given a collection of query-answer
pairs and thematically linked documents. Two
datasets from different domains are induced,1
and we identify potential pitfalls and devise
circumvention strategies. We evaluate two
previously proposed competitive models and
find that one can integrate information across
documents. However, both models struggle to
select relevant information, as providing documents guaranteed to be relevant greatly improves their performance. While the models
outperform several strong baselines, their best
accuracy reaches 42.9% compared to human
performance at 74.0% – leaving ample room
for improvement.

1

Introduction

Devising computer systems capable of answering
questions about knowledge described using text has
been a longstanding challenge in Natural Language
1

Available at http://qangaroo.cs.ucl.ac.uk

Figure 1: A sample from the W IKI H OP dataset where it
is necessary to combine information spread across multiple documents to infer the correct answer.

Processing (NLP). Contemporary end-to-end Reading Comprehension (RC) methods can learn to extract the correct answer span within a given text and
approach human-level performance (Kadlec et al.,
2016; Seo et al., 2017).
However, for existing tasks, relevant information
is often concentrated locally within a single sentence, emphasising the role of locating, matching,
and aligning information between query and support
text.2 For example, Weissenborn et al. (2017) observed that a simple binary word-in-query indicator
feature boosted the relative accuracy of a baseline
model by 27.9%.
We argue that, in order to further the ability of
machine comprehension methods to extract knowledge from text, we must move beyond a scenario
where relevant information is coherently and explicitly stated within a single document. Methods with
2

Although annotators are encouraged to pose complex
queries (Rajpurkar et al., 2016).

this capability would benefit search and Question
Answering (QA) applications where the required information cannot be found in one location. They
would also aid Information Extraction (IE) applications, such as discovering drug-drug interactions by
connecting protein interactions reported across different publications.
Figure 1 shows an example from W IKIPEDIA,
where the goal is to identify the country property
of the Hanging Gardens of Mumbai. This cannot
be inferred solely from the article about them without additional background knowledge – since the answer is not stated explicitly. However, several of
the linked articles mention the correct answer India (and other countries), but cover different topics
(e.g. Mumbai, The Arabian Sea, etc.). Finding the
answer requires multi-hop reasoning:3 figuring out
that the Hanging Gardens are located in Mumbai,
and then, from a second document, that Mumbai is a
city in India.
We define a novel RC task in which a model
should learn to answer queries by combining evidence stated across documents. We introduce a
methodology to induce datasets for this task and derive two datasets. The first, W IKI H OP, uses sets of
W IKIPEDIA articles where answers to queries about
specific properties of an entity cannot be located in
the entity’s article. In the second dataset, M ED H OP,
the goal is to establish drug-drug interactions based
on scientific findings about drugs, proteins, as well
as their interactions, found across multiple M ED LINE abstracts. For both datasets we draw upon
existing Knowledge Bases (KBs), W IKIDATA and
D RUG BANK, as ground truth, utilising distant supervision (Mintz et al., 2009) to induce the data – similar to Hewlett et al. (2016) and Joshi et al. (2017).
We establish that for 74.1% and 68.0% of the
samples, the answer can be inferred from the given
documents by a human annotator. Still, constructing multi-document datasets is challenging; we encounter and prescribe remedies for several pitfalls
associated with their assembly – for example spurious co-locations of answers and specific documents.
For both datasets we then establish several strong
baselines and evaluate the performance of two previ-

Figure 2: A bipartite graph connecting entities and documents mentioning them. The bold edges are those traversed for the first fact in the small KB on the right; yellow highlighting indicates documents in Sq and candidates in Cq . Check and cross indicate correct and false
answer candidates.

ously proposed competitive RC models (Seo et al.,
2017; Weissenborn et al., 2017). We find that one
can integrate information across documents, but neither excels at selecting relevant information from a
larger set of documents as their accuracy increases
significantly when provided only documents guaranteed to be relevant. The best model reaches 42.9%,
compared to human performance at 74.0%, indicating ample room for improvement.
In summary, our key contributions are as follows:
Firstly, proposing a cross-document multi-step RC
task, as well as a general dataset induction strategy. Secondly, assembling two datasets from different domains and identifying dataset construction
pitfalls and remedies. Thirdly, establishing multiple
baselines, including two recently proposed RC models, as well as analysing model behaviour in detail
through ablation studies.

2

Task & Dataset Construction Method

We will now formally define the multi-hop RC task,
and a generic methodology to construct multi-hop
RC datasets. Later, in Sections 3 and 4 we will
demonstrate how this method is applied in practice
by creating datasets for two different domains.
Task Formalisation A model is given a query q, a
set of supporting documents Sq , and a set of candidate answers Cq – all of which are mentioned in Sq .
The goal is to identify the correct answer a∗ ∈ Cq
by drawing on the support documents Sq . Queries
could potentially have several true answers when not

3

In contrast to Simple Question Answering (Bordes et al.,
2015a), where only a single piece of information is needed.

2

the end points corresponds to a correct answer to q.5
When traversing the graph starting at s, several
of the end points will be visited, though generally
not all; those visited define the candidate set Cq . If
however the correct answer a∗ is not among them we
discard the entire (q, a∗ ) pair. The documents visited
to reach the end points will define the support document set Sq . That is, Sq comprises chains of documents leading not only from the query subject to the
correct answer candidate, but also to type-consistent
false answer candidates.
With this methodology, relevant textual evidence
for (q, a∗ ) will be spread out across documents
along the chain connecting s and a∗ – ensuring
that multi-hop reasoning goes beyond resolving coreference within a single document. Note that including other type-consistent candidates alongside
a∗ as end points in the graph traversal – and thus
into the support documents – renders the task considerably more challenging (Jia and Liang, 2017).
Models could otherwise identify a∗ in the documents by simply relying on type consistency heuristics. It is worth pointing out that by introducing
false candidates we counterbalance a type consistency bias, in contrast to Hermann et al. (2015) and
Hill et al. (2015). We will next describe how we apply this generic dataset construction methodology in
two domains to create the W IKI H OP and M ED H OP
datasets.

constrained to rely on a specific set of support documents – e.g. queries about the parent of a certain
individual. However, in our setup each sample has
only one true answer among Cq and Sq .
Note that even though we will utilise background
information during dataset assembly, such information will not be available to a model: the document
set will be provided in random order and without
any metadata. While certainly beneficial, this would
distract from our goal of fostering end-to-end RC
methods that infer facts by combining separate facts
stated in text.
Dataset Assembly We will now describe a generic
method to construct datasets for the aforementioned
task, in a way such that finding the answer to a query
depends on multiple documents with distinct pieces
of relevant information. We assume that there exists
a document corpus D, together with a KB containing fact triples (s, r, o) – with subject entity s, relation r, and object entity o. For example, one such
fact could be (Hanging Gardens of Mumbai, country, India). We start with individual KB facts and
transform them into query-answer pairs by leaving
the object slot empty, i.e. q=(s, r, ?) and a∗ =o.
Next, we define a directed bipartite graph, where
vertices on one side correspond to documents in D,
and vertices on the other side are entities from the
KB – see Figure 2 for an example. A document node
d is connected to an entity e if e is mentioned in d,
though there may be further constraints when defining the graph connectivity. For a given (q, a∗ ) pair,
the candidates Cq and support documents Sq ⊆ D
are identified by traversing the bipartite graph using
breadth-first search; the documents visited will become the support documents Sq .
As the traversal starting point, we use the node
belonging to the subject entity s of the query q. As
traversal end points, we use the set of all entity nodes
that are type-consistent answers to q.4 Note that
whenever there is another fact (s, r, o0 ) in the KB,
i.e. a fact producing the same q but with a different
a∗ , we will not include o0 into the set of end points
for this sample. This ensures that precisely one of

3

W IKI H OP

W IKIPEDIA contains an abundance of humancurated cross-domain information and has several structured resources such as infoboxes and
W IKIDATA (Vrandečić, 2012) associated with it.
W IKIPEDIA has thus been used for a wealth of research to build datasets posing queries about a single
sentence (Morales et al., 2016; Levy et al., 2017) or
article (Yang et al., 2015a; Hewlett et al., 2016; Rajpurkar et al., 2016). However, no attempt has been
made to construct a cross-document multi-step RC
dataset based on W IKIPEDIA.
A recently proposed RC dataset is W IKI R EAD ING (Hewlett et al., 2016), where W IKIDATA tuples (item, property, answer) are aligned with

4

To determine entities which are type-consistent with the
relation r of the query, we consider all entities in the KB which
are observed as object in a fact with r as relation type – including the correct answer.

5

Here we rely on a closed-world assumption; that is, we assume that the facts in the KB state all true facts.

3

the W IKIPEDIA articles regarding their item. The
tuples define a slot filling task with the goal of predicting the answer, given an article and property.
One problem with using W IKI R EADING as an extractive RC dataset is that 54.4% of the samples
do not state the answer explicitly in the given article (Hewlett et al., 2016). However, we observed
that some of the articles accessible by following hyperlinks from the given article often state the answer,
alongside other plausible false answer candidates.
3.1

property country, this would be the set
{France, Russia, . . . }.
Graph traversal is executed up to a maximum
chain length of 3 documents. To not pose unreasonable computational constraints to RC models, examples with more than 64 different support documents
or 100 candidates are discarded, resulting in a loss
of ≈ 1% of the data.
DATA

3.2

Mitigating Dataset Biases

Dataset creation is always fraught with the risk of
inducing unintended errors and biases (Chen et al.,
2016; Schwartz et al., 2017). As Hewlett et al.
(2016) only carried out limited analysis of their
W IKI R EADING dataset, we present an analysis of
the downstream effects we observe on W IKI H OP.

Assembly

We now apply the methodology from Section 2 to
create a multi-hop dataset with W IKIPEDIA as document corpus and W IKIDATA as structured knowledge triples.6 In this setup, (item, property,
answer) W IKIDATA triples correspond to (s, r, o)
triples, and the item and property of each sample together form our query q – e.g. “(Hanging Gardens of Mumbai, country, ?)”. Similar to Yang et al.
(2015a) we only use the first paragraph of each article, since relevant information is more likely to be
stated in the beginning. Starting with all samples in
W IKI R EADING, we first remove samples where the
answer is stated explicitly in the W IKIPEDIA article
about the item.7
The bipartite graph is structured as follows:
(1) for edges from articles to entities: all articles
mentioning an entity e are connected to e; (2) for
edges from entities to articles: each entity e is only
connected to the W IKIPEDIA article about the entity.
Traversing the graph is then equivalent to iteratively
following hyperlinks to new articles about the anchor text entities.
For a given query-answer pair, the item entity
is chosen as starting point for the graph traversal. A traversal will always pass through the article about the item, since this is the only document connected from there. The end point set includes the correct answer alongside other type consistent candidate expressions, which are determined
by considering all facts belonging to W IKI R EAD ING training examples, selecting those triples with
the same property as in q and keeping their
answer expressions. As an example, for the W IKI -

Candidate Frequency Imbalance A first observation is that there is a significant bias in the answer
distribution of W IKI R EADING. For example, in the
majority of the samples the property country has
the United States of America as answer. A simple
majority class baseline would thus prove successful,
but tell us little about multi-hop reasoning. To combat this issue, we subsampled the dataset to ensure
that examples of any one particular answer candidate make up no more than 0.1% of the dataset, and
omitted articles about the United States.
Document-Answer Correlations A problem
unique to our multi-document setting is the possibility of spurious correlations between candidates
and documents induced by the graph traversal
method. In fact, if we were not to address this
issue, a model specifically designed to exploit these
regularities could achieve 74.6% accuracy (detailed
in Section 6).
Concretely, we observed that certain documents
frequently co-occur with the correct answer, independently of the query. For example, if the article about London is present among Sq in a country
query, the answer is likely to be United Kingdom, independent of the query type or entity in question. If
the article about FIFA appears, the answer is likely
to be association football. Appendix A contains a
list with several additional examples.
We designed a statistic to measure this effect
and then used it to sub-sample the dataset. The
statistic counts how often a candidate c is observed

6

We use the W IKIDATA triples present in W IKI R EADING.
We thus use a disjoint subset of W IKI R EADING compared
to Levy et al. (2017) to construct W IKI H OP.
7

4

al., 2012; Segura-Bedmar et al., 2013). However,
as shown by Peng et al. (2017), cross-sentence relation extraction increases the number of available
relations. It is thus likely that cross-document interactions would further improve recall, which is of
particular importance considering interactions that
are never stated explicitly – but rather need to be
inferred from separate pieces of evidence. The
promise of multi-hop methods is finding and combining individual observations that can suggest previously unobserved DDIs, aiding the process of
making scientific discoveries.
DDIs are caused by Protein-Protein Interaction (PPI) chains, forming biomedical pathways.
If we consider PPI chains across documents,
we find examples like in Figure 3. Here the
first document states that the drug Leuprolide
causes GnRH receptor-induced synaptic potentiations, which can be blocked by the protein
Progonadoliberin-1. The last document states that
another drug, Triptorelin, is a superagonist of the
same protein. It is therefore likely to increase the
potency of Leuprolide, describing a way in which
the two drugs interact. Besides the true interaction
there is also a false candidate Urofollitropin that,
although mentioned together with GnRH receptor
within one document, provides no textual evidence
indicating interactions with Leuprolide.

Figure 3: Sample from the M ED H OP dataset.

as the correct answer when a certain document is
present in Sq across training set samples. More formally, for a given document d and answer candidate c, let cooccurrence(d, c) denote the total count
of how often c co-occurs with d in a sample where
c is also the correct answer. We use this statistic
to filter the dataset, by discarding samples with at
least one document-candidate pair (d, c) for which
cooccurrence(d, c) > 20. This successfully mitigated the dataset bias, empirically supported by the
experiments in Section 6.

4

M ED H OP

Following the same general methodology, we next
construct a second dataset for the domain of molecular biology – a field that has been undergoing exponential growth in the number of publications (Cohen
and Hunter, 2004). The promise of applying NLP
methods to cope with this increase has led to research efforts in IE (Hirschman et al., 2005; Kim et
al., 2011) and QA for biomedical text (Hersh et al.,
2007; Nentidis et al., 2017). There is a plethora of
manually curated structured resources (Ashburner et
al., 2000; The UniProt Consortium, 2017) which can
either serve as ground truth or to induce training data
using distant supervision for NLP systems (Craven
and Kumlien, 1999; Bobic et al., 2012). Existing
RC datasets are either severely limited in size (Hersh
et al., 2007) or cover a very diverse set of query
types (Nentidis et al., 2017), complicating the application of neural models that have seen successes
for other domains (Wiese et al., 2017).
A task that has received significant attention is
detecting Drug-Drug Interactions (DDIs) (Gurulingappa et al., 2012). Existing DDI efforts have focused on explicit mentions of interactions in single sentences (Gurulingappa et al., 2012; Percha et

4.1

Assembly

We construct M ED H OP using D RUG BANK (Law
et al., 2014) as structured knowledge resource and
research paper abstracts from M EDLINE as documents. There is only a single relation type for
D RUG BANK facts, interacts with, that connects
pairs of drugs – an example of a M ED H OP query
would thus be “(Leuprolide, interacts with, ?)”. We
start by processing the 2016 M EDLINE release using the preprocessing pipeline employed for the
BioNLP 2011 Shared Task (Stenetorp et al., 2011).
We restrict the set of entities in the bipartite graph
to drugs in D RUG BANK and human proteins in
S WISS -P ROT (Bairoch et al., 2004). That is, the
graph has drugs and proteins on one side, and M ED LINE abstracts on the other.
The edge structure is as follows: (1) there is an
edge from a document to all proteins mentioned in it.
(2) there is an edge between a document and a drug,
5

if this document also mentions a protein known to
be a target for the drug according to D RUG BANK.
This edge is bidirectional, i.e. can be traversed both
ways, since there is no canonical document describing each drug – thus one can “hop” to any document
mentioning the drug and its target. (3) there is an
edge from a protein p to a document mentioning p,
but only if the document also mentions another protein p0 which is known to interact with p according to
R EACTOME (Fabregat et al., 2016). Given our distant supervision assumption, these additionally constraining requirements err on the side of precision.
As a mention, similar to Percha et al. (2012), we
consider any exact match of a name variant of a drug
or human protein in D RUG BANK or S WISS -P ROT.
For a given DDI (drug1 , interacts with, drug2 ), we
then select drug1 as the starting point for the graph
traversal. As possible end points we consider any
other drug, apart from drug1 and those interacting
with drug1 other than drug2 . Similar to W IKI H OP,
we exclude samples with more than 64 support documents and impose a maximum document length of
300 tokens plus title.8

dataset proportion

0.06
0.04
0.02
0.00

0

10

20

30

40

number of documents

50

60

Figure 4: Support documents per training sample.

W IKI H OP
M ED H OP

Train

Dev

Test

Total

43,738
1,620

5,129
342

2,451
546

51,318
2,508

Table 1: Dataset sizes for our respective datasets.

5

Dataset Analysis

Table 1 shows the dataset sizes. Note that W IKI H OP
inherits the train, development, and test set splits
from W IKI R EADING – i.e. the full dataset creation,
filtering, and sub-sampling pipeline is executed on
each set individually. Also note that sub-sampling
according to document-answer collocation (Section 3.2) significantly reduces the size of W IKI H OP
from ≈528, 000 training samples to ≈44, 000. Figure 4 illustrates the distribution of the number of
support documents per sample. W IKI H OP shows a
Poisson-like behaviour – most likely due to structural regularities in W IKIPEDIA– whereas M ED H OP
exhibits a bimodal distribution, in line with our observation that certain drugs and proteins have far
more interactions and studies associated with them.

Document Sub-sampling The bipartite graph for
M ED H OP is orders of magnitude more densely connected than for W IKI H OP. This can lead to potentially large support document sets Sq , to a degree
where it becomes computationally infeasible for a
majority of existing RC models. After the traversal
has finished, we thus subsample documents by first
adding a set of documents that connects the drug
in the query with its answer. We then iteratively
add documents to connect false candidates until we
reach the limit of 64 documents – while ensuring
that all candidates have the same number of paths
through the bipartite graph.

5.1

Mitigating Candidate Frequency Imbalance
Some drugs interact with more drugs than others
– Aspirin for example interacts with 743 other
drugs, but Isotretinoin with only 34. This leads
to similar candidate frequency imbalance issues
as with W IKI H OP – but due to its smaller size
M ED H OP is difficult to sub-sample. Nevertheless
we can successfully combat this issue by masking
entity names, detailed in Section 6.2.
8

WikiHop
MedHop

0.08

Qualitative Analysis

To establish the quality of the data and analyse potential distant supervision errors, we sample and annotate 100 samples from each development set.
W IKI H OP Table 2 lists characteristics along with
the proportion of samples that exhibit them. For
45%, the true answer either uniquely follows from
multiple texts directly or is suggested as likely. For
26%, more than one candidate is plausibly supported by the documents, including the correct an-

Same restriction as the journal PLoS ONE.

6

swer. This is often due to hypernymy, where
the appropriate level of granularity for the answer is difficult to predict – e.g. (west suffolk,
administrative entity, ?)
with candidates
suffolk and england. This is a direct consequence
of including type-consistent false answer candidates
from W IKIDATA, which can lead to questions with
several true answers. For 9% of cases a single document suffices; these samples contain a linked document that states enough information about item and
answer together. Finally, although our task is significantly more complex than most previous tasks
where distant supervision has been applied, the distant supervision assumption is only violated for 20%
of the samples – a proportion similar to previous
work (Riedel et al., 2010). These cases can either
be due to conflicting information between W IKI DATA and W IKIPEDIA (8%), e.g. when the date of
birth for a person differs between W IKIDATA and
what is stated in the W IKIPEDIA article, or because
the answer is consistent but cannot be inferred from
the support documents (12%). When answering 100
questions, the annotator knew the answer prior to
reading the documents for 9%, and produced the
correct answer after reading the document sets for
74% of the cases.

36%
9%
15%
11%
9%

Answer does not follow.
W IKIDATA/W IKIPEDIA discrepancy.

12%
8%

Table 2: Qualitiative analysis of W IKI H OP samples.

velopment and test set.9 Annotators were shown the
query-answer pair as a fact, and the chain of relevant
documents leading to the answer – similar to our
qualitative analysis of M ED H OP. They were then
instructed to answer (1) whether they knew the fact
before, (2) whether the fact follows from the texts
(with options “fact follows”, “fact is likely”, and
“fact does not follow”), and (3), whether a single or
several of the documents are required. Each sample was shown to three annotators and a majority
vote was used to aggregate the annotations. Annotators were familiar with the fact 4.6% of the time;
prior knowledge of the fact is thus not likely to be a
confounding effect on the other judgements. Interannotator agreement as measured with Fleiss’ kappa
is 0.253 in (2), and 0.281 in (3) – indicating a fair
overall agreement. Overall, 9.5% of samples have
no clear majority in (2).
Among the samples with a majority judgement,
59.8% are cases where the fact “follows”, for 14.2%
the fact was judged as “likely”, and as “not follow”
for 25.9%. This again provides good justification
for the distant supervision strategy employed during
dataset construction.
Among the samples with a majority vote for (2)
of either “follows” or “likely”, 55.9% were marked
with majority vote as requiring multiple documents
to infer the fact, and 44.1% as requiring only a single document. The latter number is larger than initially expected, given the construction of samples
through graph traversal. However, when inspecting
cases judged as “single” more closely, we observed
that many indeed provide a clear hint about the correct answer within one document, but without stating it explicitly. For example, for the fact (witold ci-

M ED H OP Since both document complexity and
number of documents per sample was significantly
larger compared to W IKI H OP (see Figure 4) it was
not feasible to ask an annotator to read all support
documents for 100 samples. We thus opted to verify the dataset quality by providing only the subset
of documents relevant to support the correct answer,
i.e. those traversed along the path reaching the answer. The annotator was asked if the answer to the
query “follows”, “is likely”, or “does not follow”,
given the relevant documents. 68% of the cases were
considered as “follows” or as “is likely”. The majority of cases violating the distant supervision assumption were errors due to the lack of a necessary
PPI in one of the connecting documents.
5.2

Unique multi-step answer.
Likely multi-step unique answer.
Multiple plausible answers.
Ambiguity due to hypernymy.
Only single document required.

Crowdsourced Human Annotation

We asked human annotators on Amazon Mechanical Turk to evaluate samples of the W IKI H OP de-

9

While desirable, crowdsourcing the same annotations for
M ED H OP is not feasible since it requires specialist knowledge.

7

chy, country of citizenship, poland) with documents
d1 : Witold Cichy (born March 15, 1986 in Wodzisaw
lski) is a Polish footballer[...] and d2 : Wodzisaw
lski[...] is a town in Silesian Voivodeship, southern
Poland[...], the information provided in d1 suffices
for a human given the background knowledge that
Polish is an attribute related to Poland, removing the
need for d2 to infer the answer.

6

candidate with the highest TF-IDF similarity score:

arg max[max TF-IDF(q + c, s)]
c∈Cq

This section describes experiments on W IKI H OP
and M ED H OP with the goal of establishing the performance of several baseline models, including recent neural RC models. We empirically demonstrate
the importance of mitigating dataset biases, probe
whether multi-step behaviour is beneficial for solving the task, and investigate if RC models can learn
to perform lexical abstraction.

arg max[max(cooccurrence(d, c))]
c∈Cq

d∈Sq

(2)

Extractive RC models: FastQA & BiDAF In
our experiments we evaluate two recently proposed
LSTM-based extractive QA models: the Bidirectional Attention Flow model (BiDAF, Seo et al.
(2017)), and FastQA (Weissenborn et al., 2017),
which have shown robust performance across several extractive QA datasets. These models predict
an answer span within a single document. In order
to apply them in a multi-document setting, we concatenate all d ∈ Sq and add document separator tokens. During training, the first answer mention in the
concatenated document serves as the gold span; at
test time we measure accuracy based on exact string
matching.
The order of the concatenated support documents
is randomised. In a preliminary experiment we
trained models using various document order permutations, but found that the performance did not
change significantly. We thus conclude that the specific order chosen does not have a major impact on
the experiments we conducted.
For BiDAF, the default hyperparameters from the
implementation of Seo et al. (2017) are used,11 with
pretrained GloVe (Pennington et al., 2014) embeddings. However, we restrict the maximum document length to 8,192 tokens and hidden size to 20,
and train for 5,000 iterations with batchsize 16 in order to fit the model into memory.12 For FastQA we
use the implementation provided by the authors, also

Models

Random Selects a random candidate; note that the
number of candidates differs between samples.
Max-mention Predicts the most frequently mentioned candidate in the support documents Sq of a
sample – randomly breaking ties.
Majority-cand.-per-query-type Predicts the candidate c ∈ Cq that was most frequently observed as
the true answer in the training set, given the query
type of q. For W IKI H OP, the query type is the property p of the query, for M ED H OP there is only the
single query type – interacts with.
TF-IDF Retrieval-based models are known to be
strong QA baselines if candidate answers are provided (Clark et al., 2016; Welbl et al., 2017). They
search for individual documents based on keywords
in the question, but typically do not combine information across documents. The purpose of this baseline is to see if it is possible to identify the correct
answer from a single document alone through lexical correlations. The model forms its prediction as
follows: For each candidate c, the concatenation of
the query q with c is fed as an OR query into the
whoosh text retrieval engine.10 It then predicts the
10

(1)

Document-cue During dataset construction we
observed that certain document-answer pairs appear
more frequently than others, to the effect that the
correct candidate is often indicated solely by the
presence of certain documents in Sq . This baseline
captures how easy it is for a model to exploit these
informative document-answer co-occurrences. It
predicts the candidate with highest score across Cq :

Experiments

6.1

s∈Sq

11

https://github.com/allenai/bi-att-flow
Given the multi-document nature of our datasets we have
an increased number of tokens per sample, thus the additional
memory requirements.
12

https://pypi.python.org/pypi/Whoosh/

8

ing with a single support document is not enough to
build a strong predictive model for both datasets.
The Document-cue baseline can predict more than
a third of samples correctly, for both datasets, even
after sub-sampling frequent document-answer pairs
in W IKI H OP. The relative strength of this and
other baselines proves to be an important issue when
designing multi-hop datasets, which we addressed
through the measures described in Section 3.2. In
Table 4 we compare the two relevant baselines on
W IKI H OP before and after applying filtering measures. The absolute strength of these baselines before filtering shows how vital addressing this issue
is: 74.6% accuracy could be reached through exploiting the cooccurrence(d, c) statistic alone. This
underlines the paramount importance of investigating and addressing dataset biases that otherwise
would confound seemingly strong RC model performance on a given dataset. The relative drop demonstrates that the measures undertaken can successfully mitigate the issue. A downside to this aggressive filtering is a significantly reduced dataset size,
which renders it infeasible for smaller datasets like
M ED H OP.
Among the two neural RC models, BiDAF is overall strongest across both datasets – this is in contrast
to the reported results for SQuAD where their performance is nearly indistinguishable. This is possibly due to the iterative latent interactions in the
BiDAF architecture: we hypothesise that these are
of increased importance for our task, where information is distributed across documents. Overall it
is worth emphasising that both FastQA and BiDAF,
which are extractive QA models, do not rely on the
candidate options Cq at all – unlike the other baselines – but predict the answer by extracting a span
from the support documents.
In the masked setup all baseline models reliant on
lexical cues fail in face of the randomised answer expressions, since the same answer option has different placeholders in different examples. Especially
on M ED H OP, where dataset sub-sampling is not a
viable option, masking proves to be a valuable alternative, effectively circumventing spurious statistical
correlations that RC models can learn to exploit.
Both neural RC models are able to largely retain
or even improve their strong performance when answers are masked: they are able to leverage the con-

with pre-trained GloVe embeddings, no characterembeddings, no maximum support length, hidden
size 50, and batch size 64 for 50 epochs.
6.2

Lexical Abstraction: Candidate Masking

The presence of lexical regularities among answers is a problem in RC dataset assembly – a
phenomenon already observed by Hermann et al.
(2015).
When comprehending a text, the correct answer
should become clear from its context – rather than
from an intrinsic property of the answer expression.
To evaluate the ability of models to rely on context
alone, we created masked versions of the datasets:
we replace any candidate expression randomly using 100 unique placeholder tokens, e.g. ”Mumbai
is the most populous city in MASK7.” Masking is
consistent within one sample, but generally different
for the same expression if it appears in another sample. This not only removes answer frequency cues,
it also removes statistical correlations between frequent answer strings and support documents. Models consequently cannot base their prediction on intrinsic properties of the answer expression, but have
to rely on the context surrounding the mentions.
6.3

Results & Discussion

Table 3 shows the experimental outcomes for W IK I H OP and M ED H OP, together with results for the
masked setting; we will first discuss the former. A
first observation is that candidate mention frequency
does not produce better predictions than a random
guess. Predicting the answer most frequently observed at training time achieves strong results: as
much as 38.8% and 58.4% on the two datasets. That
is, a simple frequency statistic together with answer
type constraints alone is a relatively strong predictor,
and the strongest overall for the “unmasked” version
of M ED H OP.
The TF-IDF retrieval baseline clearly performs
better than random for W IKI H OP, but is not very
strong overall. That is, the question tokens are helpful to detect relevant documents, but exploiting only
this information compares poorly to the other baselines. On the other hand, as no co-mention of an
interacting drug pair occurs within any single document in M ED H OP, the TF-IDF baseline performs
worse than random. We conclude that lexical match9

Model

W IKI H OP

W IKI H OP Masked

M ED H OP

M ED H OP Masked

Random
Max-mention
Majority-cand.-per-q.type
TF-IDF
Document-cue

11.5
10.6
38.8
25.6
36.7

12.2
13.9
12.0
14.4
7.4

13.9
9.5
58.4
9.0
44.9

14.1
9.2
10.4
8.8
15.2

FastQA
BiDAF

25.7
42.9

35.8
54.5

23.1
47.8

31.3
33.7

Table 3: Test accuracies for the W IKI H OP and M ED H OP datasets.

Model
Document-cue
Maj. candidate
TF-IDF
Train set size

Unfiltered

Filtered

74.6
41.2
43.8

36.7
38.8
25.6

527,773

43,738

6.4
WH

WH-GC

MH

MH-GC

BiDAF
BiDAF mask

42.9
54.5

57.9
81.2

47.8
33.7

86.4
99.3

FastQA
FastQA mask

25.7
35.8

44.5
65.3

23.1
31.3

54.6
51.8

WH

WH-rem

MH

MH-rem

BiDAF
FastQA

54.5
35.8

44.6
38.0

33.7
31.3

30.4
28.6

Table 6: Test accuracy (masked) when only documents
containing answer candidates are given (rem). WH:
W IKI H OP; MH: M ED H OP.

Table 4: Accuracy comparison for simple baseline models on W IKI H OP before and after filtering.

Model

Model

Using only relevant documents

We conducted further experiments to examine the
RC models when presented with only the relevant
documents in Sq , i.e. the chain of documents leading to the correct answer. This allows us to investigate the hypothetical performance of the models
if they were able to select and read only relevant
documents: Table 5 summarises these results. Models improve greatly in this gold chain setup with up
to 81.2% on W IKI H OP in the masked setting for
BiDAF. This demonstrates that RC models are capable of identifying the answer when few or no plausible false candidates are mentioned, which is particularly evident for M ED H OP where documents tend
to discuss only single drug candidates. On the other
hand, it also shows that the models’ answer selection
process is not robust to the introduction of unrelated
documents with type consistent candidates. Lastly,
the results indicate that learning to intelligently select relevant documents before RC may be among
the most promising directions for future model development.

Table 5: Test accuracy when only using relevant documents. GC: gold chain; WH: W IKI H OP; MH: M ED H OP.

text around the candidate expressions. To understand differences in model behaviour between W IK I H OP and M ED H OP , it is worth noting that drug
mentions in M ED H OP are normalised to a unique
single-word identifier, and performance drops under masking. In contrast, for the open-domain setting of W IKI H OP, a reduction of the answer vocabulary to 100 single-token expressions clearly helps
the model in selecting a candidate span, compared
to the multi-token candidate expressions in the unmasked setting. Overall, although both neural RC
models now clearly outperform the other baselines,
they still have large room for improvement compared to human performance at 74% for W IKI H OP.

6.5

Removing relevant documents

To investigate whether the neural RC models can
draw upon information requiring multi-step inference we designed an experiment where we discard
10

all documents from Sq that do not contain a candidate mention, including the first documents traversed. Table 6 shows the results: we can observe that performance drops across the board for
BiDAF. There is a significant drop of 3.3% on M ED H OP, but the drop for W IKI H OP is drastic at 10% –
demonstrating that BiDAF, is able to leverage crossdocuments information. FastQA shows a slight increase of 2.2% for W IKI H OP and a decrease of 2.7%
on M ED H OP. While inconclusive, it is clear that
FastQA with fewer latent interactions than BiDAF
has problems integrating cross-document information.

7

ensures that multi-step inference goes beyond resolving co-reference.
Compositional Knowledge Base Inference
Combining multiple facts is common for structured
knowledge resources which formulate facts using
first-order logic. KB inference methods include
Inductive Logic Programming (Quinlan, 1990;
Pazzani et al., 1991; Richards and Mooney, 1991)
and probabilistic relaxations to logic like Markov
Logic (Richardson and Domingos, 2006; Schoenmackers et al., 2008). These approaches suffer
from limited coverage and inefficient inference,
though efforts to circumvent sparsity have been
undertaken (Schoenmackers et al., 2008; Schoenmackers et al., 2010). A more scalable approach
to composite rule learning is the Path Ranking
Algorithm (PRA) (Lao and Cohen, 2010; Lao et al.,
2011), which performs random walks to identify
salient paths between entities. Gardner et al. (2013)
circumvent the sparsity problems in PRA by introducing synthetic links via dense latent embeddings.
Several other multi-fact inference methods based
on dense representations have been proposed, using
composition functions such as vector addition (Bordes et al., 2014), RNNs (Neelakantan et al., 2015;
Das et al., 2016), and memory networks (Jain,
2016). Another approach is the Neural Theorem
Prover (Rocktäschel and Riedel, 2017), which
uses dense rule and symbol embeddings to learn a
differentiable backward chaining algorithm.
All these previous approaches centre around
learning how to combine facts from a KB, i.e. in a
structured form with pre-defined schema. That is,
they work as part of a pipeline, and either rely on
output of a previous IE step (Banko et al., 2007), or
on direct human annotation (Bollacker et al., 2008)
which tends to be costly and biased in coverage.
However, recent neural RC methods (Seo et al.,
2017; Shen et al., 2017a) have demonstrated that
end-to-end language understanding approaches can
infer answers directly from text – sidestepping intermediate query parsing and IE steps. Our work
aims to evaluate whether end-to-end multi-step RC
models can indeed operate on raw text documents
only – while performing the kind of inference most
commonly associated with logical inference methods operating on structured knowledge.

Related Work

Related Datasets End-to-end text-based QA has
witnessed a surge in interest with the advent of largescale datasets, which have been assembled based
on F REEBASE (Berant et al., 2013; Bordes et al.,
2015b), W IKIPEDIA (Yang et al., 2015b; Rajpurkar
et al., 2016; Hewlett et al., 2016), web search
queries (Nguyen et al., 2016), news articles (Hermann et al., 2015; Onishi et al., 2016), books (Hill
et al., 2015; Paperno et al., 2016), science exams (Welbl et al., 2017), and trivia (Boyd-Graber
et al., 2012; Dunn et al., 2017). Besides TriviaQA (Joshi et al., 2017), all these datasets are confined to single documents, and RC typically does not
require a combination of multiple independent facts.
In contrast, W IKI H OP and M ED H OP are specifically designed for cross-document RC and multistep inference. There exist other multi-hop RC resources, but they are either very limited in size, such
as the FraCaS test suite, or based on synthetic language (Weston et al., 2015). Fried et al. (2015) have
demonstrated that exploiting information from other
related documents based on lexical semantic similarity is beneficial for re-ranking answers in opendomain non-factoid QA. Their method is related to
ours, but the document connections are based on the
lexical semantic similarity between words, whereas
in our approach it is based on the relation between
specific entities. TriviaQA partly involves multi-step
reasoning, but the complexity largely stems from
parsing compositional questions. Our datasets centre around compositional inference from comparatively simple queries and the cross-document setup
11

Acknowledgments

Text-Based Multi-Step Reading Comprehension
A rich collection of neural network models tailored
towards multi-step RC has been developed over the
last few years. Memory networks (Weston et al.,
2014; Sukhbaatar et al., 2015; Kumar et al., 2016)
define a generic model class that iteratively attends
over memory items defined via text, and they show
promising performance on synthetic tasks requiring
multi-step reasoning (Weston et al., 2015). One
common characteristic of neural multi-hop models
is their rich structure that enables matching and interaction between question, context, answer candidates and combinations thereof (Peng et al., 2015;
Weissenborn, 2016; Xiong et al., 2016; Liu and
Perez, 2017), which often is iterated over several
times (Sordoni et al., 2016; Mark et al., 2016; Seo
et al., 2016; Hu et al., 2017) and may contain trainable stopping mechanisms (Graves, 2016; Shen et
al., 2017b). All these methods show promise in
single-document RC, and by design should be capable of integrating multiple facts across documents.
However, thus far they have not been evaluated for a
cross-document multi-step RC task – as in this work.

We would like to thank Tim Dettmers, Pasquale
Minervini, Jeff Mitchell, and Sebastian Ruder for
several helpful comments and feedback regarding an
earlier draft of this paper. This work was supported
by an Allen Distinguished Investigator Award, a
Marie Curie Career Integration Award and an Engineering and Physical Sciences Research Council
scholarship.

References
Michael Ashburner, Catherine A. Ball, Judith A. Blake,
David Botstein, Heather Butler, J. Michael Cherry, Allan P. Davis, Kara Dolinski, Selina S. Dwight, Janan T.
Eppig, et al. 2000. Gene ontology: tool for the unification of biology. Nature Genetics, 25(1):25.
Amos Bairoch, Brigitte Boeckmann, Serenella Ferro, and
Elisabeth Gasteiger. 2004. Swiss-prot: Juggling between evolution and stability. Briefings in Bioinformatics, 5(1):39–55.
Michele Banko, Michael J. Cafarella, Stephen Soderland,
Matt Broadhead, and Oren Etzioni. 2007. Open information extraction from the web. In Proceedings of the
20th International Joint Conference on Artifical Intelligence, IJCAI’07, pages 2670–2676, San Francisco,
CA, USA. Morgan Kaufmann Publishers Inc.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy
Liang. 2013. Semantic parsing on freebase from
question-answer pairs. In Proceedings of the 2013
Conference on Empirical Methods in Natural Language Processing, pages 1533–1544.
Tamara Bobic, Roman Klinger, Philippe Thomas, and
Martin Hofmann-Apitius. 2012. Improving distantly
supervised extraction of drug-drug and protein-protein
interactions. In Proceedings of the Joint Workshop on
Unsupervised and Semi-Supervised Learning in NLP,
pages 35–43, Avignon, France, April. Association for
Computational Linguistics.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim
Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human
knowledge. In SIGMOD 08 Proceedings of the 2008
ACM SIGMOD international conference on Management of data, pages 1247–1250.
Antoine Bordes, Sumit Chopra, and Jason Weston.
2014. Question answering with subgraph embeddings.
CoRR, abs/1406.3676.
Antoine Bordes, Nicolas Usunier, Sumit Chopra, and
Jason Weston. 2015a. Large-scale simple question answering with memory networks.
CoRR,
abs/1506.02075.

Learning Search Expansion Other research addresses expanding the document set available to
a QA system, either in the form of web navigation (Nogueira and Cho, 2016), or via query
reformulation techniques, which often use neural
reinforcement learning (Narasimhan et al., 2016;
Nogueira and Cho, 2017; Buck et al., 2017). While
related, this work ultimately aims at reformulating
queries to better acquire evidence documents, and
not at answering queries through combining facts.

8

Conclusions

We have introduced a new cross-document multihop RC task, devised a generic dataset derivation
strategy and applied it to two separate domains. The
resulting datasets test RC methods in their ability to
perform composite reasoning – something thus far
limited to models operating on structured knowledge
resources. In our experiments we found that contemporary RC models can leverage cross-document information, but a sizeable gap to human performance
remains. Finally, we identified the selection of relevant document sets as the most promising direction
for future research.
12

Antoine Bordes, Nicolas Usunier, Sumit Chopra, and
Jason Weston. 2015b. Large-scale simple question answering with memory networks.
CoRR,
abs/1506.02075.
Jordan Boyd-Graber, Brianna Satinoff, He He, and Hal
Daumé, III. 2012. Besting the quiz master: Crowdsourcing incremental classification games. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and
Computational Natural Language Learning, EMNLPCoNLL ’12, pages 1290–1301, Stroudsburg, PA,
USA. Association for Computational Linguistics.
Christian Buck, Jannis Bulian, Massimiliano Ciaramita,
Andrea Gesmundo, Neil Houlsby, Wojciech Gajewski,
and Wei Wang. 2017. Ask the right questions: Active question reformulation with reinforcement learning. CoRR, abs/1705.07830.
Danqi Chen, Jason Bolton, and Christopher D. Manning.
2016. A thorough examination of the cnn/daily mail
reading comprehension task. In Proceedings of the
54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages
2358–2367, Berlin, Germany, August. Association for
Computational Linguistics.
Peter Clark, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Turney, and Daniel
Khashabi. 2016. Combining retrieval, statistics, and
inference to answer elementary science questions. In
Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI’16, pages 2580–2586. AAAI
Press.
Kevin Bretonnel Cohen and Lawrence Hunter. 2004.
Natural language processing and systems biology. Artificial intelligence methods and tools for systems biology, pages 147–173.
Mark Craven and Johan Kumlien. 1999. Constructing
biological knowledge bases by extracting information
from text sources. In Proceedings of the Seventh International Conference on Intelligent Systems for Molecular Biology, pages 77–86. AAAI Press.
Rajarshi Das, Arvind Neelakantan, David Belanger, and
Andrew McCallum. 2016. Chains of reasoning over
entities, relations, and text using recurrent neural networks. arXiv preprint arXiv:1607.01426.
Matthew Dunn, Levent Sagun, Mike Higgins, V. Ugur
Güney, Volkan Cirik, and Kyunghyun Cho. 2017.
Searchqa: A new q&a dataset augmented with context
from a search engine. CoRR, abs/1704.05179.
Antonio Fabregat, Konstantinos Sidiropoulos, Phani
Garapati, Marc Gillespie, Kerstin Hausmann, Robin
Haw, Bijay Jassal, Steven Jupe, Florian Korninger,
Sheldon McKay, Lisa Matthews, Bruce May, Marija Milacic, Karen Rothfels, Veronica Shamovsky,

Marissa Webber, Joel Weiser, Mark Williams, Guanming Wu, Lincoln Stein, Henning Hermjakob, and Peter D’Eustachio. 2016. The reactome pathway knowledgebase. Nucleic Acids Research, 44(D1):D481–
D487.
Daniel Fried, Peter Jansen, Gustave Hahn-Powell, Mihai
Surdeanu, and Peter Clark. 2015. Higher-order lexical semantic models for non-factoid answer reranking.
Transactions of the Association of Computational Linguistics, 3:197–210.
Matt Gardner, Partha Pratim Talukdar, Bryan Kisiel, and
Tom M. Mitchell. 2013. Improving learning and inference in a large knowledge-base using latent syntactic
cues. In EMNLP, pages 833–838. ACL.
Alex Graves. 2016. Adaptive computation time for recurrent neural networks. CoRR, abs/1603.08983.
Harsha Gurulingappa, Abdul Mateen Rajput, Angus
Roberts, Juliane Fluck, Martin Hofmann-Apitius, and
Luca Toldo. 2012. Development of a benchmark corpus to support the automatic extraction of drug-related
adverse effects from medical case reports. Journal of
Biomedical Informatics, 45(5):885 – 892. Text Mining and Natural Language Processing in Pharmacogenomics.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman,
and Phil Blunsom. 2015. Teaching machines to read
and comprehend. In Advances in Neural Information
Processing Systems, pages 1693–1701.
William Hersh, Aaron Cohen, Lynn Ruslen, and Phoebe
Roberts. 2007. Trec 2007 genomics track overview.
In NIST Special Publication.
Daniel Hewlett, Alexandre Lacoste, Llion Jones, Illia
Polosukhin, Andrew Fandrianto, Jay Han, Matthew
Kelcey, and David Berthelot. 2016. Wikireading:
A novel large-scale language understanding task over
wikipedia. arXiv preprint arXiv:1608.03542.
Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015. The goldilocks principle: Reading children’s books with explicit memory representations.
CoRR, abs/1511.02301.
Lynette Hirschman, Alexander Yeh, Christian Blaschke,
and Alfonso Valencia. 2005. Overview of biocreative:
critical assessment of information extraction for biology. BMC Bioinformatics, 6(1):S1, May.
Minghao Hu, Yuxing Peng, and Xipeng Qiu. 2017.
Mnemonic reader for machine comprehension. CoRR,
abs/1705.02798.
Sarthak Jain. 2016. Question answering over knowledge
base using factual memory networks. In Proceedings
of NAACL-HLT, pages 109–115.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In

13

Processing Systems (NIPS), Barcelona, Spain, December.
Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. 2009. Distant supervision for relation extraction
without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and
the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 1003–1011,
Suntec, Singapore, August. Association for Computational Linguistics.
Alvaro Morales, Varot Premtoon, Cordelia Avery, Sue
Felshin, and Boris Katz. 2016. Learning to answer
questions from wikipedia infoboxes. In Proceedings
of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1930–1935, Austin,
Texas, November. Association for Computational Linguistics.
Karthik Narasimhan, Adam Yala, and Regina Barzilay.
2016. Improving information extraction by acquiring
external evidence with reinforcement learning. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016,
Austin, Texas, USA, November 1-4, 2016, pages 2355–
2365.
Arvind Neelakantan, Benjamin Roth, and Andrew McCallum. 2015. Compositional vector space models for knowledge base completion. arXiv preprint
arXiv:1504.06662.
Anastasios Nentidis, Konstantinos Bougiatiotis, Anastasia Krithara, Georgios Paliouras, and Ioannis Kakadiaris. 2017. Results of the fifth edition of the bioasq
challenge. In BioNLP 2017, pages 48–57, Vancouver,
Canada,, August. Association for Computational Linguistics.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao,
Saurabh Tiwary, Rangan Majumder, and Li Deng.
2016.
MS MARCO: A human generated machine reading comprehension dataset.
CoRR,
abs/1611.09268.
Rodrigo Nogueira and Kyunghyun Cho. 2016. Webnav:
A new large-scale task for natural language based sequential decision making. CoRR, abs/1602.02261.
Rodrigo Nogueira and Kyunghyun Cho. 2017. Taskoriented query reformulation with reinforcement
learning. CoRR, abs/1704.04572.
Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel,
and David A. McAllester. 2016. Who did what: A
large-scale person-centered cloze dataset. In Proceedings of the 2016 Conference on Empirical Methods in
Natural Language Processing, EMNLP 2016, Austin,
Texas, USA, November 1-4, 2016, pages 2230–2235.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro

Empirical Methods in Natural Language Processing
(EMNLP).
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke
Zettlemoyer. 2017. Triviaqa: A large scale distantly
supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of
the Association for Computational Linguistics, Vancouver, Canada, July. Association for Computational
Linguistics.
Rudolf Kadlec, Martin Schmid, Ondrej Bajgar, and Jan
Kleindienst. 2016. Text understanding with the attention sum reader network. CoRR, abs/1603.01547.
Jin-Dong Kim, Yue Wang, Toshihisa Takagi, and Akinori
Yonezawa. 2011. Overview of genia event task in
bionlp shared task 2011. In Proceedings of BioNLP
Shared Task 2011 Workshop.
Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer,
Ishaan Gulrajani James Bradbury, Victor Zhong, Romain Paulus, and Richard Socher. 2016. Ask me
anything: Dynamic memory networks for natural language processing. International Conference on Machine Learning.
Ni Lao and William W Cohen. 2010. Relational retrieval using a combination of path-constrained random walks. Machine learning, 81(1):53–67.
Ni Lao, Tom Mitchell, and William W Cohen. 2011.
Random walk inference and learning in a large scale
knowledge base. In Proceedings of the Conference on
Empirical Methods in Natural Language Processing,
pages 529–539. Association for Computational Linguistics.
Vivian Law, Craig Knox, Yannick Djoumbou, Tim Jewison, An Chi Guo, Yifeng Liu, Adam Maciejewski, David Arndt, Michael Wilson, Vanessa Neveu,
Alexandra Tang, Geraldine Gabriel, Carol Ly, Sakina
Adamjee, Zerihun T. Dame, Beomsoo Han, You Zhou,
and David S. Wishart. 2014. Drugbank 4.0: shedding new light on drug metabolism. Nucleic Acids Research, 42(D1):D1091–D1097.
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading
comprehension. In Proceedings of the 21st Conference on Computational Natural Language Learning
(CoNLL 2017), pages 333–342, Vancouver, Canada,
August. Association for Computational Linguistics.
Fei Liu and Julien Perez. 2017. Gated end-to-end memory networks. In Proceedings of the 15th Conference
of the European Chapter of the Association for Computational Linguistics, EACL 2017, Valencia, Spain,
April 3-7, 2017, Volume 1: Long Papers, pages 1–10.
Neumann Mark, Pontus Stenetorp, and Sebastian Riedel.
2016. Learning to reason with adaptive computation.
In Interpretable Machine Learning for Complex Systems at the 2016 Conference on Neural Information

14

Pezzelle, Marco Baroni, Gemma Boleda, and Raquel
Fernández. 2016. The lambada dataset: Word prediction requiring a broad discourse context. arXiv
preprint arXiv:1606.06031.
M. J. Pazzani, C. A. Brunk, and G. Silverstein. 1991.
A knowledge-intensive approach to learning relational
concepts. In Proc. of the Eighth International Workshop on Machine Learning, pages 432–436, Evanston,
IL.
Baolin Peng, Zhengdong Lu, Hang Li, and Kam-Fai
Wong. 2015. Towards neural network-based reasoning. CoRR, abs/1508.05508.
Nanyun Peng, Hoifung Poon, Chris Quirk, Kristina
Toutanova, and Wen-tau Yih. 2017. Cross-sentence
n-ary relation extraction with graph lstms. Transactions of the Association for Computational Linguistics,
5:101–115.
Jeffrey Pennington, Richard Socher, and Christopher D.
Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543.
Bethany Percha, Yael Garten, and Russ B Altman. 2012.
Discovery and explanation of drug-drug interactions
via text mining. In Pacific symposium on biocomputing, page 410. NIH Public Access.
J. R. Quinlan. 1990. Learning logical definitions from
relations. MACHINE LEARNING, 5:239–266.
P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. 2016.
Squad: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP).
B. L. Richards and R. J. Mooney. 1991. First-order
theory revision. In Proc. of the Eighth International Workshop on Machine Learning, pages 447–
451, Evanston, IL.
Matthew Richardson and Pedro Domingos.
2006.
Markov logic networks. Mach. Learn., 62(1-2):107–
136, February.
Sebastian Riedel, Limin Yao, and Andrew McCallum.
2010. Modeling relations and their mentions without labeled text. In Proceedings of the 2010 European
Conference on Machine Learning and Knowledge Discovery in Databases: Part III, ECML PKDD’10,
pages 148–163, Berlin, Heidelberg. Springer-Verlag.
Tim Rocktäschel and Sebastian Riedel. 2017. End-toend differentiable proving. In Advances in Neural
Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2017,
December 4-9, 2017, Long Beach, California, United
States, volume abs/1705.11040.
Stefan Schoenmackers, Oren Etzioni, and Daniel S.
Weld. 2008. Scaling textual inference to the web.
In EMNLP ’08: Proceedings of the Conference on

Empirical Methods in Natural Language Processing,
pages 79–88, Morristown, NJ, USA. Association for
Computational Linguistics.
Stefan Schoenmackers, Oren Etzioni, Daniel S. Weld,
and Jesse Davis. 2010. Learning first-order horn
clauses from web text. In Proceedings of the 2010
Conference on Empirical Methods in Natural Language Processing, EMNLP ’10, pages 1088–1098,
Stroudsburg, PA, USA. Association for Computational
Linguistics.
Roy Schwartz, Maarten Sap, Ioannis Konstas, Leila
Zilles, Yejin Choi, and Noah A. Smith. 2017. The
effect of different writing tasks on linguistic style: A
case study of the roc story cloze task. In Proceedings
of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 15–25, Vancouver, Canada, August. Association for Computational
Linguistics.
Isabel Segura-Bedmar, Paloma Martı́nez, and Marı́a Herrero Zazo. 2013. Semeval-2013 task 9 : Extraction of
drug-drug interactions from biomedical texts (ddiextraction 2013). In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2:
Proceedings of the Seventh International Workshop on
Semantic Evaluation (SemEval 2013), pages 341–350,
Atlanta, Georgia, USA, June. Association for Computational Linguistics.
Min Joon Seo, Hannaneh Hajishirzi, and Ali Farhadi.
2016. Query-regression networks for machine comprehension. CoRR, abs/1606.04582.
Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and
Hannaneh Hajishirzi. 2017. Bidirectional attention
flow for machine comprehension. In The International
Conference on Learning Representations (ICLR).
Yelong Shen, Po-Sen Huang, Jianfeng Gao, and Weizhu
Chen. 2017a. Reasonet: Learning to stop reading in
machine comprehension. In Proceedings of the 23rd
ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’17, pages
1047–1055, New York, NY, USA. ACM.
Yelong Shen, Po-Sen Huang, Jianfeng Gao, and Weizhu
Chen. 2017b. Reasonet: Learning to stop reading in
machine comprehension. In Proceedings of the 23rd
ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’17, pages
1047–1055, New York, NY, USA. ACM.
Alessandro Sordoni, Phillip Bachman, and Yoshua Bengio. 2016. Iterative alternating neural attention for
machine reading. CoRR, abs/1606.02245.
Pontus Stenetorp, Goran Topić, Sampo Pyysalo, Tomoko
Ohta, Jin-Dong Kim, and Jun’ichi Tsujii. 2011.
Bionlp shared task 2011: Supporting resources. In
Proceedings of BioNLP Shared Task 2011 Workshop,

15

pages 112–120, Portland, Oregon, USA, June. Association for Computational Linguistics.
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al.
2015. End-to-end memory networks. In Advances in
neural information processing systems, pages 2440–
2448.
The UniProt Consortium. 2017. Uniprot: the universal protein knowledgebase. Nucleic Acids Research,
45(D1):D158–D169.
Denny Vrandečić. 2012. Wikidata: A new platform
for collaborative data collection. In Proceedings of
the 21st International Conference on World Wide Web,
WWW ’12 Companion, pages 1063–1064, New York,
NY, USA. ACM.
Dirk Weissenborn, Georg Wiese, and Laura Seiffe. 2017.
Fastqa: A simple and efficient neural architecture for
question answering. CoRR, abs/1703.04816.
Dirk Weissenborn. 2016. Separating answers from
queries for neural reading comprehension. CoRR,
abs/1607.03316.
Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017.
Crowdsourcing multiple choice science questions. In
Proceedings of the Third Workshop on Noisy Usergenerated Text, Copenhagen, Denmark. Association
for Computational Linguistics.
Jason Weston, Sumit Chopra, and Antoine Bordes. 2014.
Memory networks. arXiv preprint arXiv:1410.3916.
Jason Weston, Antoine Bordes, Sumit Chopra, and
Tomas Mikolov. 2015. Towards ai-complete question answering: A set of prerequisite toy tasks. CoRR,
abs/1502.05698.
Georg Wiese, Dirk Weissenborn, and Mariana L. Neves.
2017. Neural question answering at bioasq 5b. CoRR,
abs/1706.08568.
Caiming Xiong, Victor Zhong, and Richard Socher.
2016. Dynamic coattention networks for question answering. CoRR, abs/1611.01604.
Yi Yang, Wen-tau Yih, and Christopher Meek. 2015a.
Wikiqa: A challenge dataset for open-domain question
answering. In Proceedings of the 2015 Conference on
Empirical Methods in Natural Language Processing,
pages 2013–2018, Lisbon, Portugal, September. Association for Computational Linguistics.
Yi Yang, Wen-tau Yih, and Christopher Meek. 2015b.
Wikiqa: A challenge dataset for open-domain question
answering. In Proceedings of the 2015 Conference on
Empirical Methods in Natural Language Processing,
pages 2013–2018, Lisbon, Portugal, September. Association for Computational Linguistics.

16

WikiHop
MedHop

WikiHop

dataset proportion

dataset proportion

0.05
0.04
0.03
0.02
0.01
0.00

0

10

20

30

40

number of candidates

50

E

Appendix: Document-Cue examples

Appendix: Gold Chain Examples

Appendix: Query Types

Table 9 gives an overview over the 25 most frequent
query types in W IKI H OP and their relative proportion in the dataset. Overall, the distribution across
the 277 query types follows a power law.

D

0.02

0

100

200

300

400

document length

500

600

Apprendix: Document Lengths

Figure 6 shows the distribution of document lengths
for both datasets. Note that the document lengths
in W IKI H OP correspond to the lengths of the first
paragraphs of W IKIPEDIA articles. M ED H OP on the
other hand reflects the length of research paper abstracts, which are generally longer.

Table 8 shows examples of document gold chains
in W IKI H OP. Note that their lengths differ, with a
maximum of 3 documents.

C

0.04

Figure 6: Histogram for document lengths in W IKI H OP
and M ED H OP.

Table 7 shows examples of answers and articles
which frequently appear together in W IKI H OP before filtering.

B

0.06

0.00

60

Figure 5: Histogram for the number of candidates per
sample in W IKI H OP.

A

0.08

Appendix: Candidate and Document
statistics

Figure 5 shows a histogram with the number of candidates per sample in W IKI H OP, and the distribution shows a slow but steady decrease. For M ED H OP, the vast majority of samples has 9 candidates,
which is due to the way documents are selected up
until a maximum of 64 documents is reached. Very
few samples have less than 9 candidates, and samples would have far more false candidates if more
than 64 support documents were included.
17

Answer a

Wikipedia article d

Count Prop.

united states of
america

A U.S. state is a constituent political entity of the United States of America.

68,233 12.9%

united kingdom

England is a country that is part of the United Kingdom.

54,005 10.2%

taxon

In biology, a species (abbreviated sp., with the plural form species abbreviated spp.) is the basic unit of biological classification and a taxonomic
rank.

40,141 7.6%

taxon

A genus (pl. genera) is a taxonomic rank used in the biological classification

38,466 7.3%

united kingdom

The United Kingdom of Great Britain and Northern Ireland, commonly
known as the United Kingdom (UK) or Britain, is a sovereign country in
western Europe.

31,071 5.9%

taxon

Biology is a natural science concerned with the study of life and living organisms, including their structure, function, growth, evolution, distribution,
identification and taxonomy.

27,609 5.2%

united kingdom

Scotland [...] is a country that is part of the United Kingdom and covers the
northern third of the island of Great Britain.

25,456 4.8%

united kingdom

Wales [...] is a country that is part of the United Kingdom and the island of
Great Britain.

21,961 4.2%

united kingdom

London [...] is the capital and most populous city of England and the United
Kingdom, as well as the most populous city proper in the European Union.

21,920 4.2%

...

...

...

united states of
america

Nevada (Spanish for ”snowy”; see pronunciations) is a state in the Western,
Mountain West, and Southwestern regions of the United States of America.

18,215 3.4%

...

...

...

italy

The comune [...] is a basic administrative division in Italy, roughly equivalent to a township or municipality.

8,785 1.7%

...

...

...

A town is a human settlement larger than a village but smaller than a city.

5,092 1.0%

...

...

...

people’s republic of china

Shanghai [...] often abbreviated as Hu or Shen, is one of the four directcontrolled municipalities of the People’s Republic of China.

3,628 0.7%

human
ment

settle-

Table 7:
Examples with largest cooccurrence(d, c) statistic, before filtering. The Count column states
cooccurrence(d, c); the last column states the corresponding relative proportion of training samples (total 527,773).

18

Query: (the big broadcast of 1937, genre, ?)
Answer: musical film
Text 1: The Big Broadcast of 1937 is a 1936 Paramount Pictures production directed by Mitchell Leisen,
and is the third in the series of Big Broadcast movies. The musical comedy stars Jack Benny, George Burns,
Gracie Allen, Bob Burns, Martha Raye, Shirley Ross [...]
Text 2: Shirley Ross (January 7, 1913 March 9, 1975) was an American actress and singer, notable for her
duet with Bob Hope, ”Thanks for the Memory” from ”The Big Broadcast of 1938”[...]
Text 3: The Big Broadcast of 1938 is a Paramount Pictures musical film featuring W.C. Fields and Bob
Hope. Directed by Mitchell Leisen, the film is the last in a series of ”Big Broadcast” movies[...]
Query: (cmos, subclass of, ?)
Answer: semiconductor device
Text 1: Complementary metal-oxide-semiconductor (CMOS) [...] is a technology for constructing integrated circuits. [...] CMOS uses complementary and symmetrical pairs of p-type and n-type metal oxide
semiconductor field effect transistors (MOSFETs) for logic functions. [...]
Text 2: A transistor is a semiconductor device used to amplify or switch electronic signals[...]
Query: (raik dittrich, sport, ?)
Answer: biathlon
Text 1: Raik Dittrich (born October 12, 1968 in Sebnitz) is a retired East German biathlete who won two
World Championships medals. He represented the sports club SG Dynamo Zinnwald [...]
Text 2: SG Dynamo Zinnwald is a sector of SV Dynamo located in Altenberg, Saxony[...] The main sports
covered by the club are biathlon, bobsleigh, luge, mountain biking, and Skeleton (sport)[...]
Query: (minnesota gubernatorial election, office contested, ?)
Answer: governor
Text 1: The 1936 Minnesota gubernatorial election took place on November 3, 1936. Farmer-Labor Party
candidate Elmer Austin Benson defeated Republican Party of Minnesota challenger Martin A. Nelson.
Text 2: Elmer Austin Benson [...] served as the 24th governor of Minnesota, defeating Republican Martin
Nelson in a landslide victory in Minnesota’s 1936 gubernatorial election.[...]
Query: (ieee transactions on information theory, publisher, ?)
Answer: institute of electrical and electronics engineers
Text 1: IEEE Transactions on Information Theory is a monthly peer-reviewed scientific journal published by the IEEE Information Theory Society [...] the journal allows the posting of preprints [...]
Text 2: The IEEE Information Theory Society (ITS or ITSoc), formerly the IEEE Information Theory
Group, is a professional society of the Institute of Electrical and Electronics Engineers (IEEE) [...]
Query: (country of citizenship, louis-philippe fiset, ?)
Answer: canada
Text1: Louis-Philippe Fiset [...] was a local physician and politician in the Mauricie area [...]
Text2: Mauricie is a traditional and current administrative region of Quebec. La Mauricie National Park is
contained within the region, making it a prime tourist location. [...]
Text3: La Mauricie National Park is located near Shawinigan in the Laurentian mountains, in the Mauricie
region of Quebec, Canada [...]
Table 8: Examples of document gold chains in W IKI H OP. Article titles are boldfaced, the correct answer is underlined.

19

Query Type

Proportion in Dataset

instance of
located in the administrative territorial entity
occupation
place of birth
record label
genre
country of citizenship
parent taxon
place of death
inception
date of birth
country
headquarters location
part of
subclass of
sport
member of political party
publisher
publication date
country of origin
languages spoken or written
date of death
original language of work
followed by
position held

10.71 %
9.50 %
7.28 %
5.75 %
5.27 %
5.03 %
3.45 %
3.16 %
2.46 %
2.20 %
1.84 %
1.70 %
1.52 %
1.43 %
1.40 %
1.36 %
1.29 %
1.16 %
1.06 %
0.92 %
0.92 %
0.90 %
0.85 %
0.82 %
0.79 %

Top 25
Top 50
Top 100
Top 200

72.77 %
86.42 %
96.62 %
99.71 %

Table 9: The 25 most frequent query types in W IKI H OP alongside their proportion in the training set.

20

