We present KB LRN, a framework for end-toend learning of knowledge base representations from latent, relational, and numerical features. KB LRN integrates feature types with
a novel combination of neural representation
learning and probabilistic product of experts
models. To the best of our knowledge, KB LRN
is the first approach that learns representations of knowledge bases by integrating latent, relational, and numerical features. We
show that instances of KB LRN outperform existing methods on a range of knowledge base
completion tasks. We contribute a novel data
sets enriching commonly used knowledge base
completion benchmarks with numerical features. We have made the data sets available for
further research1 . We also investigate the impact numerical features have on the KB completion performance of KB LRN.

1

Introduction

The importance of knowledge bases (KBs) for AI systems has been demonstrated numerous times. KBs provide ways to organize, manage, and retrieve structured
data and allow AI system to perform reasoning. In recent years, KBs have been playing an increasingly crucial role in several application domains. Purely logical
representations of knowledge bases have a long history
in AI [24]. However, they suffer from being inefficient
and brittle. Inefficient because the computational complexity of reasoning is exponential in the worst case and,
therefore, the time required by a reasoner highly unpredictable. Brittle because a purely logical KB requires
a large set of logical rules that are handcrafted and/or
1

https://github.com/nle-ml/mmkb

Murasaki Shikibu

Gotoh Museum
dIn

Abstract

Mathias Niepert
NEC Labs Europe
Heidelberg, Germany
mathias.niepert@neclab.eu

locate

Alberto Garcı́a-Durán
NEC Labs Europe
Heidelberg, Germany
alberto.duran@neclab.eu

Tokyo

hasArtAbout
bornIn

arXiv:1709.04676v2 [] 12 Mar 2018

KB LRN: End-to-End Learning of Knowledge Base Representations with
Latent, Relational, and Numerical Features

captialOf
loc

ate
Latitude: 35.65
dIn
Area: 2,2
Avg. salary: 520,060

Japan

Latitude: 36.21
Area: 378
Avg. salary: 482,163

Sensō-ji
Figure 1: A small part of a knowledge base.

mined. These problems are even more pressing in applications whose environments are changing over time.
Motivated by these shortcomings, there has been a flurry
of work on combining logical and statistical approaches
to build systems capable of reasoning over and learning from incomplete structured data. Most notably, the
statistical relational learning community has proposed
numerous formalisms that combine logic and probability [21]. These formalism are able to address the learning
problem and make the resulting AI systems more robust
to missing data and missing rules. Intuitively, logical formulas act as relational features and the probability of a
possible world is determined by a sufficient statistic for
the values of these features. These approaches, however,
are in in most cases even less efficient because logical
inference is substituted with probabilistic inference.
More recently, the research community has focused on
efficient machine learning models that perform well on
restricted tasks such as link prediction in KBs. Examples are knowledge base factorization and embedding
approaches [2, 19, 10, 18] and random-walk based ML
models [14, 6]. The former learn latent features for the
entities and relations in the knowledge base and use those
to perform link prediction. The latter explore specific relational features such as path types between two entities

and train a machine learning model for link prediction.
With this work, we propose KB LRN, a novel approach to
combining relational, latent (learned), and numerical features, that is, features that can take on a large or infinite
number of real values. The combination of various features types is achieved by integrating embedding-based
learning with probabilistic models in two ways. First,
we show that modeling numerical features with radial
basis functions is beneficial and can be integrated in an
end-to-end differentiable learning system. Second, we
propose a probabilistic product of experts (PoE) [12] approach to combine the feature types. Instead of training
the PoE with contrastive divergence, we approximate the
partition function with a negative sampling strategy. The
PoE approach has the advantage of being able to train the
model jointly and end-to-end.
The paper is organized as follows. First, we discuss relational, latent, and numerical features. Second, we describe KB LRN. Third, we present empirical evidence
that instances of KB LRN outperform state of the art
methods for KB completion. We also investigate in detail
under what conditions numerical features are beneficial.

2

Relational, Latent, and Numerical
Features

We assume that the facts of a knowledge base (KB) are
given as a set of triples of the form (h, r, t) where h and
t are the head and tail entities and r is a relation type.
Figure 1 depicts a small fragment of a KB with relations
and numerical features. KB completion is the problem of
answering queries of the form (?, r, t) or (h, r, ?). While
the proposed approach can be generalized to more complex queries, we focus on completion queries for the sake
of simplicity. We now discuss the three features types
used in KB LRN and motivate their utility for knowledge
base completion. How exactly we extract features from
a given KB is described in the experimental section.
2.1

Relational Features

Each relational features is given as a logical formula
which is evaluated in the KB to determine the feature’s
value. For instance, the formula ∃x (A, bornIn, x) ∧
(x, capitalOf, B) corresponds to a binary feature which
is 1 if there exists a path of that type from entity A to entity B, and 0 otherwise. These features are often used in
relational models [27, 20] and random-walk based models such as PRA and SFE [14, 6]. In this work, we use
relational paths of length one and two and use the rule
mining approach A MIE + [4]. We detail the generation of
the relational features in the experimental section. For a

pair of entities (h, t), we denote the feature vector computed based on a set of relational features by r(h,t) .
2.2

Latent Features

Numerous embedding methods for KBs have been proposed in recent years [19, 2, 10, 18]. Embedding methods provide fixed-size vector representations (embeddings) for all entities in the KB. In the simplest of cases,
relations are modeled as translations in the entity embedding space [2]. We incorporate typical embedding learning objectives into KB LRN and write eh and et to refer
to an embedding of a head entity and a tail entity, respectively. The advantages of latent feature models are their
computational efficiency and their ability to learn latent
entity types suitable for downstream ML tasks without
hand-crafted or mined logical rules.
2.3

Numerical Features

Numerical features are entity features whose values can
take on a very large or infinite number of real values.
To the best of our knowledge, there does not exists a
principled approach that integrates numerical features
into a relational ML model for KB completion. This
is surprising, considering that numerical data is available in almost all existing large-scale KBs. The assumption that numerical data is helpful for KB completion
tasks is reasonable. For several relations types the differences between the head and tail are characteristic of
the relation itself. For example, while the mean difference of birth years is 0.4 for the Freebase relation
/people/marriage/spouse, it is 32.4 for the relation /person/children. These observations motivate specifically the use of differences of numerical feature values. Taking the difference has the advantage that
even if a numerical feature is not distributed according
to a normal distribution (e.g., birth years in a KB), the
difference is often normally distributed. This is important as we need to fit simple parametric distributions to
the sparse numerical data. We detail the fully automated
extraction and generation of the numerical features in the
experimental section.

3

KB LRN: Learning End-To-End Joint
Representations for Knowledge Bases

With KB LRN we aim to provide a framework for endto-end learning of KB representations. Since we want
to combine different feature types (relational, latent
or learned, and numerical) we need to find a suitable
method for integrating the respective submodels, one per
feature type. We propose a product of experts (PoE)
approach where one expert is trained for each (relation

1

2

( * ). ( * ).
eh et w

r

eh et' wr

...

softmax(

Product Of Experts

...

N+1

1

+

.

... ( * ).
eh et'' wr

2

.

wrrel

r(h,t)

)

L

...

wrrel

r(h,t')

...

N+1

.

categorical cross-entropy loss
1

+

.

wrrel

2

wrnum

ϕ(n(h,t))

r(h,t'')

.

wrnum

N+1

...
...

ϕ(n(h,t'))

.

wrnum

ϕ(n(h,t''))

Figure 2: An illustration of an instance of KB LRN. For every relation, there is a seperate expert for each of the different
feature types. The entities t0 and t00 are two of N randomly sampled entities. The scores of the various submodels are
added and normalized with a softmax function. A categorical cross-entropy loss is applied to the normalized scores.
type, feature type) pair. We extend the product of experts
approach in two novel ways. First, we create dependencies between the experts by sharing the parameters of the
entity embedding model across relation types. By doing
this, we combine a probabilistic model with a model that
learns vector representations from discrete and numerical data. Second, while product of experts are commonly
trained with contrastive divergence [12], we train it with
negative sampling and a cross-entropy loss.

is equivalent to the exponential of the D IST M ULT [31]
scoring function. With KB LRN we can use any of the
existing KB embedding scoring functions.

In general, a PoE’s probability distribution is
Q
m fm (d | θm )
p(d | θ1 , ..., θn ) = P Q
,
c
m fm (c | θm )

where c indexes all possible triples.

where d is a data vector in a discrete space, θm are the
parameters of individual model m, fm (d | φm ) is the
value of d under model m, and the c’s index all possible
vectors in the data space. The PoE model is now trained
to assign high probability to observed data vectors.
In the KB context, the data vector d is always a triple
d = (h, r, t) and the objective is to learn a PoE that
assigns high probability to true triples and low probabilities to triples assumed to be false. If (h, r, t) holds in the
KB, the pair’s vector representations are used as positive
training examples. Let d = (h, r, t). We can now define
one individual expert f(r,F) (d | φ(r,F) ) for each (relation
type r, feature type F) pair as follows
r

f(r,L) (d | θ(r,L) ) = exp((eh ∗ et ) · w )

r
f(r,R) (d | θ(r,R) ) = exp r(h,t) · wrel


r
f(r,N) (d | θ(r,N) ) = exp φ n(h,t) · wnum
and

The probability for triple d = (h, r, t) of the PoE model
is now
Q
F∈{R,L,N} f(r,F) (d | θ(r,F) )
,
p(d | θ1 , ..., θn ) = P Q
c
F∈{R,L,N} f(r,F) (c | θ(r,F) ))

For numerical features, an activation function should fire
when the difference of values is in a specific range. For
example, we want the activation to be high when the
difference of the birth years between a parent and its
child is close to 32.4 years. Commonly used activation functions such as sigmoid or tanh are not suitable
here, since they saturate whenever they exceed a certain threshold. For each relation r and the dn corresponding relevant numerical features, therefore, we apply a radial basis function over the differences of values
(1)
(dn )
φ(n(h,t) ) = [φ(n(h,t) ), . . . , φ(n(h,t)
)], where



(i)



φ n(h,t) = exp 

(i)

−||n(h,t) − ci ||22
σi2


.

This results in the RBF kernel being activated whenever
the difference of values is close to the expected value
ci . We discuss and evaluate several alternative strategies
for incorporating numerical features in the experimental
section.

f(r0 ,F) (d | θ(r0 ,F) ) = 1 for all r0 6= r and F ∈ {L, R, N},
where ∗ is the elment-wise product, · is the dot product,
r
r
wr , wrel
, wnum
are the parameter vectors for the latent,
relational, and numerical features corresponding to the
relation r, and φ is the radial basis function (RBF) applied element-wise to n(h,t) . Note that f(r,L) (d | θ(r,L) )

3.1

Learning

Product of experts are usually trained with contrastive divergence (CD) [12]. Contrastive divergence relies on an
approximation of the gradient of the log-likelihood using
a short Markov chain started at the current seen example.

The advantage of CD is that the partition function, that is,
the denominator of the probability p, which is intractable
to compute, does not need to be approximated. Due to
the parameterization of the PoE we have defined here,
however, it is not possible to perform CD since there is
no way to sample a hidden state given a triple d. Hence,
instead of using CD, we approximate the partition function by performing negative sampling.
The logarithmic loss for the given training triples T is
defined as
X
L=−
log p(t | θ1 , ..., θn ).
t∈T

To fit the PoE to the training triples, we follow the derivative of the log likelihood of each observed triple d ∈ T
under the PoE
∂ log p(d | θ1 , ..., θn ) ∂ log fm (d | θm )
=
∂θm
∂θ
P mQ
∂ log c m fm (c | θm )
−
∂θm
Now, to approximate the intractable second term of the
right hand side of the above equation, we generate for
each triple d = (h, r, t) a set E consisting of N triples
(h, r, t0 ) by sampling exactly N entities t0 uniformly at
random from the set of all entities. The term
P Q
∂ log c m fm (c | θm )
∂θm
is then approximated by the term
P
Q
∂ log c∈E m fm (c | θm )
.
∂θm
Analogously for the head of the triple. This is often referred to as negative sampling. Figure 2 illustrates the
KB LRN framework.

4

Related Work

A combination of latent and relational features has been
explored by Toutanova et al. [27, 28]. There, a weighted
combination of two independently learned models, a
latent feature model [31] and a model fed with a binary vector reflecting the presence of paths of length
one between the head and tail, is proposed. These
simple relational features aim at capturing association
strengths between pairs of relationships (e.g. contains
and contained by). [22] proposed a method that learns
implicit associations between pairs of relations in addition to a latent feature model in the context of relation extraction. [7] modifies the path ranking algorithm (PRA) [14] to incorporate latent representations

into models based on random walks in KBs. [6] extracted relational features other than paths to better capture entity type information. There are a number of recent approaches that combine relational and latent representations by incorporating known logical rules into the
embedding learning formulation [23, 9, 16]. Despite its
simplicity, KB LRN’s combination of relational and latent representations significantly outperforms all these
approaches.
There exists additional work on combining various types
of KB features. [17] proposed a modification of the
well-known tensor factorization method RESCAL [19],
called ARE, which adds a learnable matrix that weighs
a set of metrics (e.g. Common Neighbors) between pairs
of entities; [5] proposed a combination of latent features,
aiming to take advantage of the different interaction patterns between the elements of a triple.
KB LRN is different to these approaches in that (i) we incorporate numerical features for KB completion, (ii) we
propose a unifying end-to-end learning framework that
integrates arbitrary relational, latent, and numerical features.

5

Experiments

We conducted experiments on six different knowledge
base completion data sets. Primarily, we wanted to understand for what type of relations numerical features are
helpful and what input representation of numerical features achieves the best results. An additional objective
was the comparison to state of the art methods.
5.1

Datasets

We conducted experiments on six different data sets:
FB15k, FB15k-237, FB15k-num, FB15k-237-num,
WN18, and FB122. FB15k [2] and Wordnet (WN) [1]
are knowledge base completion data sets commonly used
in the literature. The FB15k data set is a representative
subset of the Freebase knowledge base. WN18 represents lexical relations between word senses. The two
data sets are being increasingly criticized for the frequent
occurrence of reverse relations causing simple relational
baselines to outperform most embedding-based methods [27]. For these reasons, we also conducted experiments with FB15k-237 a variant of FB15k where reverse
relations have been removed [27]. FB122 is a subset of
FB15k focusing on relations pertaining to the topics of
“people”, “location”, and “sports.” In previous work, a
set of 47 logical rules was created for FB122 and subsequently used in experiments for methods that take logical
rules into account [9, 16].

Data set
Entities
Relation types
Training triples
Validation triples
Test triples
Relational features

FB15k
14,951
1,345
483,142
50,000
59,071
90,318

FB15k-num
14,951
1,345
483,142
5,156
6,012
90,318

FB15k-237
14,541
237
272,115
17,535
20, 466
7,834

FB15k-237-num
14,541
237
272,115
1,058
1,215
7,834

WN18
40,943
18
141,442
5,000
5,000
14

FB122
9,738
122
91,638
9,595
11,243
47

Table 1: Statistics of the data sets.
n(h,t)
sign
RBF

MR
231
121

MRR
29.7
31.4

Hits@1
20.1
21.2

Hits@10
50.1
52.3

Table 2: KB LRN for two possible input representations
of numerical features for FB15k-237-num.
The main objective of this paper is to investigate the impact of incorporating numerical features. Hence, we created two additional data set by removing those triples
from FB15k’s and FB15k-237’s validation and test sets
where numerical features are never used for the triples’
relation type. Hence, the remaining test and validation
triples lead to completion queries where the numerical
features under consideration are potentially used. We refer to these new data sets as FB15k-num and FB15k-237num. A similar methodology can be followed to evaluate
the performance on a different set of numerical features.
We extracted numerical data from the 1.9 billion triple
Freebase RDF dump by mining triples that associate entities to literals of some numerical type. For example,
the relation /location/geocode/latitude maps
entities to their latitude. We performed the extraction for
all entities in FB15k but only kept a numerical feature
if at least 5 entities had values for it. This resulted in
116 different numerical features and 12,826 entities for
which at least one of the numerical features had a value.
On average each entity had 2.3 numerical features with
a value. The numerical features will be published. Since
numerical data is not available for Wordnet, we do not
perform experiments with numerical features for variants
of this KB.
Each data set contains a set of triples that are known to
be true (usually referred to as positive triples). Statistics of the data sets are provided in Table 1. Since the
identifiers for entities and relations have been changed in
FB13 [26], we could not extract numerical features for
the data set and excluded it from the experiments.
5.2

General Set-up

We evaluated the different methods by their ability to
answer completion queries of the form (h, r, ?) and
(?, r, t). For queries of the form (h, r, ?), we replaced
the tail by each of the KB’s entities in turn, sorted the

triples based on the scores or probabilities returned by
the different methods, and computed the rank of the correct entity. We repeated the same process for the queries
of type (?, r, t). We follow the filtered setting described
in [2] which removes correct triples that are different to
the target triple from the ranked list. The mean of all
computed ranks is the Mean Rank (lower is better) and
the fraction of correct entities ranked in the top n is called
hits@n (higher is better). We also computer the Mean
Reciprocal Rank (higher is better) which is an evaluation
measure for rankings that is less susceptible to outliers.
We conduct experiments with the scoring function of
D IST M ULT [31] which is an application of parallel factor analysis to multi-relational graphs. For a review on
parallel factor analysis we refer the reader to [11]. We
validated the embedding size of KB LRN from the values
{100, 200} for all experiments. These values are used in
most of the literature on KB embedding methods. For all
other embedding methods, we report the original results
from the literature or run the authors’ original implementation. For FB15k and FB15k-237, the results for DistMult, Complex, and R-GCN+ are taken from [25]; results for the relational baseline Node+LinkFeat are taken
from [27]; results for ConvE are taken from [3] and results for TransE were obtained by running the authors’
implementation. For WN18-rules and FB122-all, the results for TransE, TransH, TransR, and KALE are taken
from [9], and results for ComplEx and ASR-ComplEx
are taken from [16]. All methods were tuned for each of
the respective data sets.
For KB LRN we used A DAM [13] for parameter learning
in a mini-batch setting with a learning rate of 0.001, the
categorical cross-entropy as loss function and the number
of epochs was set to 100. We validated every 5 epochs
and stopped learning whenever the MRR (Mean Reciprocal Rank) values on the validation set decreased. The
batch size was set to 512 and the number N of negative samples to 500 for all experiments. We use the abbreviations KBsuffix to refer to the different instances of
KB LRN. suffix is a combination of the letters L (Latent),
R ( R elational) and N ( N umerical) to indicate the inclusion
of each of the three feature types.

T RANS E
D IST M ULT
C OMPL E X
N ODE +L INK F EAT
R-GCN+
C ONV E

MR
51
64

MRR
44.3
65.4
69.2
82.2
69.6
74.5

FB15k
Hits@1
25.1
54.6
59.9
60.1
67.0

KB L
KB R
KB LR

69
628
45

77.4
78.7
79.0

71.2
75.6
74.2

KB LN
KB RN
KB LRN

66
598
44

78.3
78.7
79.4

72.6
75.6
74.8

FB15k-237
Hits@10
MR
MRR Hits@1
76.3
214
25.1
14.5
82.4
19.1
10.6
84.0
20.1
11.2
87.0
23.7
84.2
24.8
15.3
87.3
330
30.1
22.0
without numerical features
87.6
231
30.1
21.4
84.3
2518
18.0
12.8
87.3
231
30.6
22.0
with numerical features
87.8
229
30.4
22.0
84.2
3303
18.2
13.0
87.5
209
30.9
21.9

Hits@10
46.3
37.6
38.8
36.0
41.7
45.8
47.5
28.5
48.2
47.0
28.7
49.3

Table 3: Results (filtered setting) for KB LRN and state of the art approaches.

T RANS E
D IST M ULT

MR
25
39

FB15k-num
MRR Hits@1
34.7
5.5
72.6
62.1

KB L
KB R
KB LR

39
399
28

72.6
84.7
85.3

62.1
81.6
80.3

KB LN
KB RN
KB LRN

32
68
25

73.6
84.0
85.9

63.0
80.6
81.0

FB15k-237-num
Hits@10
MR
MRR Hits@1 Hits@10
79.9
158
21.8
10.41
45.6
89.7
195
26.4
16.4
47.3
without numerical features
89.7
195
26.4
16.4
47.3
90.1
3595
23.6
17.8
36.1
92.4
232
29.3
19.7
49.2
with numerical features
90.7
122
28.6
17.9
51.6
90.0
600
26.1
19.3
39.7
92.9
121
31.4
21.2
52.3

Table 4: Results (filtered) on the data sets where the test and validation sets are comprised of those triples whose type could potentially benefit from numerical features.

T RANS E
T RANS H
T RANS R
K ALE - PRE
K ALE - JOINT
C OMPL E X
ASR-C OMPL E X
KB L
KB R
KB LR

MR
537
7113
588

MRR
45.3
56.0
51.4
53.2
66.2
94.2
94.2
80.8
72.0
93.6

WN18+rules[9]
Hits@3 Hits@5
79.1
89.1
80.0
86.1
69.7
77.5
86.4
91.9
85.5
90.1
94.7
95.0
94.7
95.0
92.5
93.7
72.1
72.1
94.5
94.8

Hits@10
93.6
90.0
84.3
94.4
93.0
95.1
95.1
94.7
72.1
95.1

MR
117
2018
113

MRR
48.0
46.0
40.1
52.3
52.3
64.1
69.8
69.5
54.7
70.2

FB122-all[9]
Hits@3 Hits@5
58.9
64.2
53.7
59.1
46.4
52.4
61.7
66.2
61.2
66.4
67.3
69.5
71.7
73.6
74.6
77.2
54.7
54.7
74.0
77.0

Hits@10
70.2
66.0
59.3
71.8
72.8
71.9
75.7
80.0
54.7
79.7

Table 5: Results (filtered setting) for KB completion benchmarks where logical rules are provided.
5.3

Automated Generation of Relational and
Numerical Features

For the data sets FB15k, FB15k-237, and their numerical versions, we used all relational paths of length one
and two found in the training data as relational features.
These correspond to the formula types (h, r, t) (1-hop)
and ∃x (h, r1 , x) ∧ (x, r2 , t) (2-hops). We computed
these relational paths with A MIE + [4], a highly efficient

system for mining logical rules from knowledge bases.
We used the standard settings of A MIE + with the exception that the minimal head support was set to 1. With
these settings, A MIE + returns horn rules of the form
body ⇒ (x, r, y) that are present for at least 1% of the
triples of the form (x, r, y). For each relation r, we used
the body of those rules where r occurs in the head as
r’s relational path features. For instance, given a rule
such as (x, r1 , z), (z, r2 , y) ⇒ (x, r, y), we introduce

Relation
capital of
spouse of
influenced by

KB LR
MRR H@10
5.7
13.6
4.4
0.0
7.3
20.9

KB LRN
MRR H@10
14.6
18.2
7.9
0.0
9.9
26.8

Table 6: MRR and hits@10 results (filtered) for KB LRN
with and without numerical features in FB15k-237. Results improve for relations where the difference of the
relevant features is approximately normal (see Figure 3).
the relational feature ∃x (h, r1 , x) ∧ (x, r2 , t) for the relation r. For the data sets WN18 and FB122, we used
the set of logical formulas previously used in the literature [9]. Using the same set of relational features allows
us to compare KB LRN with existing approaches that incorporate logical rules into the embedding learning objective [9, 16].
For each relation r we only included a numerical feature if, in at least τ = 90% of training triples, both the
head and the tail had a value for it. This increases the
likelihood that the feature is usable during test time. For
τ = 90% there were 105 relations in FB15k for which at
least one numerical feature was included during learning,
and 33 relations in FB15k-237. With the exception of the
RBF parameters, all network weights are initialized following [8]. The parameters of KB LRN’s RBF kernels
P
(i)
1
are initialized and fixed to ci = |T|
(h,r,t)∈T n(h,t) ,
where T is the set of training triples (h, r, t) for the re(i)
(i)
lationq
r for which both nh and nt have a value; and
P
(i)
1
2
σi = |T|
(h,r,t)∈T (n(h,t) − ci ) .
5.4

Representations of Numerical Features

We experimented with different strategies for incorporating raw numerical features. For the difference of feature values the simplest method is the application of the
sign function. For a numerical attribute i, the activation is either 1 or −1 depending on whether the differ(i)
(i)
ence nh − nt is positive or negative. For a more
nuanced representation of differences of numerical features, a layer of RBF kernels is a suitable choice since
the activation is here highest in a particular range of input values. The RBF kernel might not be appropriate,
however, in cases where the underlying distribution is not
normal.
To evaluate different input representations, we conducted
experiments with KB LRN on the FB15k-237-num data
set. Table 2 depicts the KB completion performance of
two representation strategies for the difference of head
and tail values. Each row corresponds to one evaluated
strategy. “sign” stands for applying the sign function to

capital_of

10 0 10

Diff. of latitude

spouse

influenced_by

30 0 30

Diff. of birth year

400 0 400

Diff. of birth year

Figure 3: Histograms and fitted RBFs for three representative relations and numerical features.
the difference of numerical feature values. RBF stands
for using an RBF kernel layer for the differences of numerical feature values. All results are for the FB15k237-num test triples.
The RBF kernels outperform the sign functions significantly. This indicates that the difference of feature values is often distributed normally and that having a region
of activation is beneficial. Given these results, we use
the RBF input layer for n(h,t) for the remainder of the
experiments.
5.5

Comparison to State of the Art

For the standard benchmark data sets FB15k and FB15k237, we compare KB LRN with T RANS E, D IST M ULT,
C OMPL E X [29], R-GCN+ [25], and ConvE [3].
Table 3 lists the KB completion results. KB LRN is competitive with state of the art knowledge base completion methods on FB15k and significantly outperforms all
other methods on the more challenging FB15k-237 data
set. Since the fraction of triples that can potentially benefit from numerical features is very small for FB15k, the
inclusion of numerical features is only slightly beneficial. For FB15k-237, however, the numerical features
significantly improve the results.
For the numerical versions of FB15k and FB15k-237, we
compared KB LRN to TransE and DistMult. Table 4 lists
the results for the KB completion task on these data sets.
KB LRN significantly outperforms the KB embedding approaches. The positive impact of including the numerical
features is significant.
For the data sets WN18-rules and FB122-all we compared KB LRN to KB embedding methods TransE,
TransR [15], TransH [30], and ComplEx [29] as well as
state of the are approaches for incorporating logical rules
into the learning process. The experimental set-up is consistent with that of previous work. Table 5 lists the results
for the KB completion task on these data sets. KB LRN
combining relational and latent representations significantly outperforms existing approaches on FB122 with

KB LR
KB LRN

O NE
42
60

MR
M ANY
201
135

O NE
74.9
74.2

MRR
M ANY
21.0
22.8

Table 7: MR and MRR results (filtered) on FB15k-237num based on the cardinality of the test queries.
exactly the same set of rules. This provides evidence
that KB LRN’s strategy to combine latent and relational
features is effective despite its simplicity relative to existing approaches. For WN18+rules, KB LRN is competitive with ComplEx, the best performing method on this
data set. In addition, KB LRN’s performance improves
significantly when relational and latent representations
are combined. In contrast, ASR-C OMPL E X is not able
to improve the results of ComplEx, its underlying latent
representation.
5.6

The Impact of Numerical Features

The integration of numerical features improves
KB LRN’s performance significantly. We performed
several additional experiments so as to gain a deeper
understanding of the impact numerical features have.
Table 6 compares KB LRN’s performance with and
without integrating numerical features on three relations. The performance of the model with numerical
features is clearly superior for all three relationships
(capital of, spouse and influenced by). Figure 3 shows the normalized histograms for the values
n(h,t) for these relations. We observe the differences of
feature values are approximately normal.
Following previous work [2], we have classified each test
query of FB15k-237-num as either O NE or M ANY, depending on the whether one or many entities can complete that query. For the queries labeled O NE the model
without numerical features shows a slightly worse performance with respect to the model that makes use of
them, whereas for the queries labeled M ANY, KB LRN
significantly outperforms KB LR in both MR and MRR.
A well-known finding is the lack of completeness of
FB15k and FB15k-237. This results in numerous cases
where the correct entity for a completion query is not
contained in the ground truth (neither in the training, nor
test, nor validation data set). This is especially problematic for queries where a large number of entities are correct completions. To investigate the actual benefits of
the numerical features we carried out the following experiment: We manually determined all correct completions for the query (USA, /location/contains, ?).
We ended up with 1619 entities that correctly complete

T RANS E
KB LR
KB LRN

AUC-PR
0.837
0.913
0.958

MR
231
94
43

MRR
26.5
66.8
70.8

Table 8: AUC-PR, MR, and MRR results for the completion query (USA, /location/contains, ?).
the query. FB15k-237 contains only 954 of these correct
completions. With a complete ground truth, we can now
use the precision-recall area under the curve (PR-AUC)
metric to evaluate KB completion methods [19, 17, 5]. A
high PR-AUC represents both high recall and high precision. Table 8 lists the results for the different methods.
KB LRN with numerical features consistently and significantly outperformed all other approaches.

6

Conclusion

We introduced KB LRN, a class of machine learning
models that, to the best of our knowledge, is the first proposal aiming at integrating relational, latent, and continuous features of KBs into a single framework. KB LRN
outperforms state of the art KB completion methods on a
range of data sets. We show that the inclusion of numerical features is beneficial for KB completion tasks.

References
[1] A. Bordes, X. Glorot, J. Weston, and Y. Bengio.
A semantic matching energy function for learning with multi-relational data. Machine Learning,
94(2):233–259, 2014.
[2] A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, and O. Yakhnenko. Translating embeddings
for modeling multi-relational data. In Advances
in Neural Information Processing Systems, pages
2787–2795, 2013.
[3] T. Dettmers, P. Minervini, P. Stenetorp, and
S. Riedel. Convolutional 2d knowledge graph embeddings. arXiv preprint arXiv:1707.01476, 2017.
[4] L. Galárraga, C. Teflioudi, K. Hose, and F. M.
Suchanek. Fast rule mining in ontological knowledge bases with amie+. The VLDB Journal,
24(6):707–730, 2015.
[5] A. Garcia-Duran, A. Bordes, N. Usunier, and
Y. Grandvalet. Combining two and three-way embeddings models for link prediction in knowledge
bases. arXiv preprint arXiv:1506.00999, 2015.

[6] M. Gardner and T. M. Mitchell. Efficient and
expressive knowledge base completion using subgraph feature extraction. In EMNLP, pages 1488–
1498, 2015.
[7] M. Gardner, P. P. Talukdar, J. Krishnamurthy, and
T. Mitchell. Incorporating vector space similarity
in random walk inference over knowledge bases. In
EMNLP, 2014.
[8] X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks.
In AISTATS, volume 9, pages 249–256, 2010.
[9] S. Guo, Q. Wang, L. Wang, B. Wang, and L. Guo.
Jointly embedding knowledge graphs and logical
rules. In EMNLP, pages 192–202, 2016.
[10] K. Guu, J. Miller, and P. Liang. Traversing knowledge graphs in vector space. In EMNLP, 2015.
[11] R. A. Harshman and M. E. Lundy. Parafac: Parallel
factor analysis. Computational Statistics & Data
Analysis, 18(1):39–72, 1994.
[12] G. E. Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771–1800, 2002.
[13] D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,
2014.
[14] N. Lao, T. Mitchell, and W. W. Cohen. Random
walk inference and learning in a large scale knowledge base. In EMNLP, pages 529–539, 2011.
[15] Y. Lin, Z. Liu, M. Sun, Y. Liu, and X. Zhu. Learning entity and relation embeddings for knowledge
graph completion. In AAAI, pages 2181–2187,
2015.
[16] P. Minervini, T. Demeester, T. Rocktäschel, and
S. Riedel. Adversarial sets for regularised neural
link predictors. In UAI, 2017.
[17] M. Nickel, X. Jiang, and V. Tresp. Reducing the
rank in relational factorization models by including observable patterns. In NIPS, pages 1179–1187,
2014.
[18] M. Nickel, K. Murphy, V. Tresp, and
E. Gabrilovich.
A review of relational machine learning for knowledge graphs. Proceedings
of the IEEE, 104(1):11–33, 2016.
[19] M. Nickel, V. Tresp, and H.-P. Kriegel. A three-way
model for collective learning on multi-relational
data. In ICML, pages 809–816, 2011.

[20] M. Niepert. Discriminative gaifman models. In
NIPS, pages 3405–3413, 2016.
[21] L. D. Raedt, K. Kersting, S. Natarajan, and
D. Poole. Statistical relational artificial intelligence: Logic, probability, and computation. Synthesis Lectures on Artificial Intelligence and Machine Learning, 10(2):1–189, 2016.
[22] S. Riedel, L. Yao, A. McCallum, and B. M. Marlin. Relation extraction with matrix factorization
and universal schemas. 2013.
[23] T. Rocktäschel, S. Singh, and S. Riedel. Injecting
logical background knowledge into embeddings for
relation extraction. In HLT-NAACL, pages 1119–
1129, 2015.
[24] S. Russell and P. Norvig. Artificial Intelligence: A
Modern Approach. Prentice Hall Press, 3rd edition,
2009.
[25] M. Schlichtkrull, T. Kipf, P. Bloem, R. v. d. Berg,
I. Titov, and M. Welling. Modeling relational data
with graph convolutional networks. arXiv preprint
arXiv:1703.06103, 2017.
[26] R. Socher, D. Chen, C. D. Manning, and A. Y. Ng.
Reasoning with neural tensor networks for knowledge base completion. In NIPS, pages 926–934,
2013.
[27] K. Toutanova and D. Chen. Observed versus latent features for knowledge base and text inference.
In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality, pages 57–66, 2015.
[28] K. Toutanova, D. Chen, P. Pantel, H. Poon,
P. Choudhury, and M. Gamon. Representing text
for joint embedding of text and knowledge bases.
In EMNLP, pages 1499–1509, 2015.
[29] T. Trouillon, J. Welbl, S. Riedel, E. Gaussier, and
G. Bouchard. Complex embeddings for simple link
prediction. In ICML, pages 2071–2080, 2016.
[30] Z. Wang, J. Zhang, J. Feng, and Z. Chen. Knowledge graph embedding by translating on hyperplanes. In AAAI, pages 1112–1119, 2014.
[31] B. Yang, W.-t. Yih, X. He, J. Gao, and
L. Deng. Learning multi-relational semantics using neural-embedding models. arXiv preprint
arXiv:1411.4072, 2014.

