Invariant components of synergy, redundancy, and unique
information among three variables
Giuseppe Pica

∗1

, Eugenio Piasini1 , Daniel Chicharro1,2 , and Stefano Panzeri

†1

1

arXiv:1706.08921v1 [] 27 Jun 2017

Neural Computation Laboratory, Center for Neuroscience and Cognitive
Systems @UniTn, Istituto Italiano di Tecnologia, Rovereto (TN) 38068, Italy
2
Department of Neurobiology, Harvard Medical School, Boston, MA 02115, USA

Abstract
In a system of three stochastic variables, the Partial Information Decomposition
(PID) of Williams and Beer dissects the information that two variables (sources)
carry about a third variable (target) into nonnegative information atoms that describe redundant, unique, and synergistic modes of dependencies among the variables. However, the classification of the three variables into two sources and one
target limits the dependency modes that can be quantitatively resolved, and does
not naturally suit all systems. Here, we extend the PID to describe trivariate modes
of dependencies in full generality, without introducing additional decomposition axioms or making assumptions about the target/source nature of the variables. By
comparing different PID lattices of the same system, we unveil a finer PID structure
made of seven nonnegative information subatoms that are invariant to different target/source classifications and that are sufficient to construct any PID lattice. This
finer structure naturally splits redundant information into two nonnegative components: the source redundancy, which arises from the pairwise correlations between
the source variables, and the non-source redundancy, which does not, and relates
to the synergistic information the sources carry about the target. The invariant
structure is also sufficient to construct the system’s entropy, hence it characterizes
completely all the interdependencies in the system.

1

Introduction

Shannon’s mutual information [1] provides a well established, widely applicable tool to
characterize the statistical relationship between two stochastic variables. Larger values
of mutual information correspond to a stronger relationship between the instantiations
of the two variables in each single trial. Whenever we study a system with more than
two variables, the mutual information between any two subsets of the variables still
quantifies the statistical dependencies between these two subsets; however, many scientific questions in the analysis of complex systems require a finer characterization of how
all variables simultaneously interact [2, 3, 4, 5, 6]. For example, two of the variables,
A and B, may carry either redundant or synergistic information about a third variable
∗
†

giuseppe.pica@iit.it
stefano.panzeri@iit.it

1

C [7, 8, 9], but considering the value of the mutual information I((A, B) : C) alone
is not enough to distinguish these qualitatively different information-carrying modes.
To achieve this finer level of understanding, recent theoretical efforts have focused on
decomposing the mutual information between two subsets of variables into more specific
information components (see e.g. [6, 10, 11, 12]). Nonetheless, a complete framework
for the information-theoretic analysis of multivariate systems is still lacking.
Here we consider the analysis of trivariate systems, which is complex enough to
present the fundamental challenges of going beyond bivariate analyses, and yet simple
enough to provide hints about how these challenges might be addressed. Characterizing
the fine structure of the interactions among three stochastic variables can improve the
understanding of many interesting problems across different disciplines [13, 14, 15, 16].
For example, in the study of neural information processing many important questions
can be cast as trivariate information analyses. Determining quantitatively how two
neurons encode information about an external sensory stimulus [7, 8, 9] requires describing the dependencies between the stimulus and the activity of the two neurons.
Determining how the stimulus information carried by a neural response relates to the
animal’s behaviour [17, 18, 19] requires the analysis of the simultaneous three-wise dependencies among the stimulus, the neural activity and the subject’s behavioral report.
More generally, a thorough understanding of even the simplest information-processing
systems would require the quantitative description of all different ways two inputs carry
information about one output [20].
In systems where legitimate assumptions can be made about which variables act as
sources of information and which variable acts as the target of information transmission, the partial information decomposition (PID) [10] provides an elegant framework
to decompose the mutual information that one or two (source) variables carry about
the third (target) variable into a finer lattice of redundant, unique and synergistic information atoms. However, in many systems the a priori classification of variables into
sources and target is arbitrary, and limits the description of the distribution of information within the system [21]. Furthermore, even when one classification is adopted, the
PID atoms do not characterize completely all the possible modes of information sharing
between the sources and the target. For example, two sources can carry redundant information about the target irrespective of the strength of the correlations between them
and, as a consequence, the PID redundancy atom can be larger than zero even if the
sources have no mutual information [22, 3, 23]. Hence, the value of the PID redundancy
measure cannot distinguish how the correlations between two variables contribute to the
information that they share about a third variable.
In this paper, we address these limitations by extending the PID framework without
introducing further axioms or assumptions about the trivariate structure to analyze.
We compare the atoms from the three possible PID lattices that are induced by the
three possible choices for the target variable in the system. By tracking how the PID
information modes change across different lattices, we move beyond the partial perspective intrinsic to a single PID lattice and unveil the finer structure common to all PID
lattices. We find that all lattices can be fully described in terms of a unique minimal
set of seven information-theoretic quantities, that is invariant to different classifications
of the variables.
The first result of this approach is the identification of two nonnegative subatomic
2

components of the redundant information that any pair of variables carries about the
third variable. The first component, that we name source redundancy (SR), quantifies
the part of the redundancy which arises from the correlations of the sources. The second
component, that we name non-source redundancy (N SR), quantifies the part of the
redundancy which is not related to the source correlations. Interestingly, we find that
whenever the non-source redundancy is larger than zero then also the synergy is larger
than zero. The second result is that the minimal set induces a unique nonnegative
decomposition of the full joint entropy H(X, Y, Z) of the system. This allows us to
dissect completely the distribution of information of any trivariate system in a general
way that is invariant with respect to the source/target classification of the variables. To
illustrate the additional insights of this new approach, we finally apply our framework
to paradigmatic examples, including discrete and continuous probability distributions.
These applications confirm our intuitions and clarify the practical usefulness of the finer
PID structure.

2

Preliminaries and state of the art

Williams and Beer proposed an influential axiomatic decomposition of the mutual information I(X : (Y, Z)) that two stochastic variables Y, Z (the sources) carry about a
third variable X (the target) into the sum of four nonnegative atoms [10]:
• SI(X : {Y ; Z}), which is the information about the target that is shared between
the two sources (the redundancy);
• U I(X : {Y \Z}) and U I(X : {Z\Y }), which are the separate pieces of information
about the target that can be extracted from one of the sources, but not from the
other;
• CI(X : {Y ; Z}), which is the complementary information about the target that is
only available when both of the sources are jointly observed (the synergy).
This construction is commonly known as the Partial Information Decomposition
(PID). Sums of subsets of the four PID atoms provide the classical mutual information
quantities between each of the sources and the target, I(X : Y ) and I(X : Z), and the
conditional mutual information quantities whereby one of the sources is the conditioned
variable, I(X : Z|Y ) and I(X : Y |Z). Such relationships are displayed with a color code
in Fig. 1.
The PID decomposition of Ref. [10] is based upon a number of axioms that do not
determine univocally the value of the four PID atoms. The specific measure proposed
in Ref. [10] to calculate such values has been questioned as it can lead to unintuitive
results [22], and thus many attempts have been devoted to finding alternative measures [24, 22, 25, 26, 27, 28] compatibly with an extended number of axioms, such as
the identity axiom proposed in [22]. Other work has studied in more detail the lattice
structure that underpins the PID, indicating the duality between information gain and
information loss lattices [12]. Even though there is no consensus on how to build partial
information decompositions in systems with more than two sources, for trivariate systems the measures of redundancy, synergy and unique information defined in Ref. [11]
3

I(X:Y)
UI(X: {Y\Z})

Y

X

SI(X: {Y;Z})

Z

UI(X: {Z\Y})
I(X:Z)

(a)

CI(X: {Y;Z})

(b)

Figure 1: The Partial Information Decomposition as defined by Williams and Beer’s
axioms [10]. a) The mutual information of the sources Y, Z about the target X is
decomposed into four atoms: the redundancy SI(X : {Y ; Z}), the unique informations
U I(X : {Y \Z}), U I(X : {Z\Y }), and the synergy CI(X : {Y ; Z}). The colored
rectangles represent the linear equations that relate the four PID atoms to four Shannon
information quantities. b) An exploded view of the allotment of information between
the sources Y, Z and the target X: each PID atom of panel a) corresponds to a thick
dotted line, while the colored stripes represent the two pairwise mutual informations
between each of the sources and the target (with the same color code as in a)). Each of
the mutual informations splits into the sum of the redundancy with its corresponding
unique information. The circuit-diagram symbol for the XOR operation is associated to
the synergistic component CI(X{Y ; Z}) only for illustration, as XOR is often taken as
a paradigmatic example of synergistic interaction between variables.
have not yet been questioned and have found wide acceptance (in this paper, we will
in fact make use of these measures when a concrete implementation of the PID will be
required).
However, even in the trivariate case there are open problems regarding the understanding of the PID atoms in relation to the interdependencies within the system. First,
Harder et al. [22] pointed out that the redundant information shared between the sources
about the target can intuitively arise from the following two qualitatively different modes
of three-wise interdependence:
• the source redundancy, which is redundancy which ’must already manifest itself in
the mutual information between the sources’1 ;
• the mechanistic redundancy, which can be larger than zero even if there is no
mutual information between the sources.
As pointed out by Harder and colleagues [22], a more precise conceptual and formal
separation of these two kinds of redundancy still needs to be achieved, and presents
fundamental challenges. The very notion that two statistically independent sources can
nonetheless share information about a target was not captured by some earlier definitions
1

Note that Ref. [22] interchangeably refers to the sources as ’inputs’: we will discuss this further in
Section 4 when addressing the characterization of source redundancy.

4

of redundancy [24, 25]. Further, several studies [23, 29] described the property that
the PID measures of redundancy can be positive even when there are no correlations
between the sources as undesired. On a different note, other authors [20] pointed out
that the two different notions of redundancy can define qualitatively different modes of
information processing in (neural) input-output networks.
Other issues were recently pointed out by James and Crutchfield [21], who indicated
that the very definition of the PID lattice prevents its use as a general tool for assessing
the full structure of trivariate (let alone multi-variate) statistical dependencies. In particular, Ref. [21] considered dyadic and triadic systems, which underlie quite interesting
and common modes of multivariate interdependencies. They showed that, even though
the PID atoms are among the very few measures that can distinguish between the two
kinds of systems, a PID lattice cannot allot the full joint entropy H(X, Y, Z) of either
system. The decomposition of the joint entropy in terms of information components
that reflect qualitatively different interactions within the system has also been subject
of recent research [30, 27].
In summary, the PID framework, in its current form, does not yet provide a satisfactorily fine and complete description of the distribution of information in trivariate
systems. The PID atoms do assess trivariate dependencies better than Shannon’s measures, but they cannot quantify interesting finer interdependencies within the system,
such as the source redundancy that the sources share about the target. In addition, they
are limited to describing the dependencies between the chosen sources and target, thus
enforcing a certain perspective on the system that does not naturally suit all systems.

3

More PID diagrams unveil finer structure in the PID
framework

To address the open problems described above, we begin by pointing out the feature of
the PID lattice that underlies all the issues in the characterization of trivariate systems
outlined in Sec. 2. As we illustrate in Fig. 1a, while a single PID diagram involves the
mutual information quantities that one or both of the sources (in the figure, Y and
Z) carry about the target X, it does not contain the mutual information between the
sources I(Y : Z) and their conditional mutual information I(Y : Z|X). This precludes
the characterization of source redundancy with a single PID diagram, as it prevents any
comparison between the elements of the PID and I(Y : Z). Moreover, it also signals
that a single PID lattice cannot account for the total entropy H(X, Y, Z) of the system.
These considerations suggest that the inability of the PID framework to provide a
complete information-theoretic description of trivariate systems is not a consequence of
the axiomatic construction underlying the PID lattice. Instead, it follows from restricting the analysis to the limited perspective on the system that is enforced by classifying
the variables into sources and target when defining a PID lattice. We thus elaborate
that significant progress can be achieved, without the addition of further axioms or assumptions to the PID framework, if one just considers, alongside the PID diagram in
Fig. 1a, the other two PID diagrams that are induced respectively by labeling Y or Z as
the target in the system. When considering the resulting three PID diagrams (Fig. 2),
the previously missing mutual information I(Y : Z) between the original sources of the

5

Figure 2: The three possible PIDs of a trivariate probability distribution p(x, y, z) that
follow from the three possible choices for the target variable: on the left the target is X,
in the middle it is Y and on the right it is Z. The coloured rectangles highlight the linear
relationships between the twelve PID atoms and the six Shannon information quantities.
Note that the orientations of the PIDs for I(X : (Y, Z)) and I(Z : (X, Y )) are rotated
with respect to the PID for I(Y : (X, Z)) to highlight their reciprocal relations, as will
become more apparent in Fig. 4.
left-most diagram is now decomposed into PID atoms of the middle and the right-most
diagrams in Fig. 2, and the same happens with I(Y : Z|X).
In the following we take advantage of this shift in perspective to resolve the finer
structure of the PID diagrams and, at the same time, to generalize its descriptive
power going beyond the current limited framework, where only the information that
two (source) variables carry about the third (target) variable is decomposed. More
specifically, even though the PID relies on setting a partial point of view about the system, we will show that describing how the PID atoms change when we systematically
rotate the choice of the PID target variable effectively overcomes the limitations intrinsic
to one PID alone.

3.1

The relationship between PID diagrams with different target selections

To identify the finer structure underlying all the PID diagrams in Fig. 2, we first focus
on the relationships between the PID atoms of two different diagrams, with the goal of
understanding how to move from one perspective on the system to another. The key
observation here is that, for each pair of variables in the system, their mutual information
and their conditional mutual information given the third variable appear in two of the
PID diagrams. This imposes some constraints relating the PID atoms in two different

6

a

b

Figure 3: (a) The relationships between the PID atoms from two diagrams with different
target selections. When we swap a target and a source, the differences in the amount
of redundancy, synergy, and unique information with respect to the third variable are
not independent due to equations of the type of Eqs. 1 and 2. Here, we consider
the PID diagram of I(X : (Y, Z)) (left) and I(Y : (X, Z)) (right) from Fig. 2, under
the assumption that SI(Y : {X; Z}) ≤ SI(X : {Y ; Z}). (b) The reversible pieces of
Z

Z

Z

information RSI(X ↔ Y ) (orange block), RCI(X ↔ Y ) (gray block) and RU I(X ↔ Y )
(magenta block) contribute to the same kind of atom across the two PID diagrams. The
Z

irreversible piece of information IRSI(X ← Y ) (light blue block) contributes to different
kinds of atom across the two PID diagrams. The remaining two unique information
atoms (black dots) are not constrained by equations of the type of Eqs. 1 and 2.
diagrams. For example, if we consider I(X : Y ) and I(X : Y |Z), we find that:
I(X : Y ) = SI(X : {Y ; Z}) + U I(X : {Y \Z}) = SI(Y : {X; Z}) + U I(Y : {X\Z}),
(1)
I(X : Y |Z) = CI(X : {Y ; Z}) + U I(X : {Y \Z}) = CI(Y : {X; Z}) + U I(Y : {X\Z}),
(2)
where the first and second equality in each equation result from the decomposition of
I(X : (Y, Z)) (left-most diagram in Fig. 2) and I(Y : (X, Z)) (middle diagram in Fig. 2),
respectively. From Eq. 1 we see that, when the roles of a target and a source are reversed
(here, the roles of X and Y ), the difference in redundancy is the opposite of the difference
in unique information with respect to the other source (here, Z). Similarly, Eq. 2 shows
that the difference in synergy is the opposite of the difference in unique information
with respect to the other source. Combining these two equalities, we also see that the
difference in redundancy is equal to the difference in synergy. Therefore, the equalities
impose relationships across some PID atoms appearing in two different diagrams.
These relationships are depicted in Figure 3. The eight PID atoms appearing in the
two diagrams can be expressed in terms of only six subatoms, due to the constraints
of the form of Eqs. 1 and 2. In particular, to select the smallest nonnegative pieces of

7

information resulting from the constraints, we define:
Z

RSI(X ↔ Y ) := min[SI(X : {Y ; Z}), SI(Y : {X; Z})],
Z

RCI(X ↔ Y ) := min[CI(X : {Y ; Z}), CI(Y : {X; Z})],

(3a)
(3b)

Z

RU I(X ↔ Y ) := min[U I(X : {Y \Z}), U I(Y : {X\Z})].

(3c)

The above terms are called the Reversible Shared Information of X and Y considering Z
Z
(RSI(X ↔ Y ); the orange block in Fig. 3), the Reversible Complementary Information
Z

of X and Y considering Z (RCI(X ↔ Y ); the gray block in Fig. 3), and the Reversible
Z

Unique Information of X and Y considering Z (RU I(X ↔ Y ); the magenta block in
Fig. 3). The attribute reversible highlights that, when we reverse the roles of target
and source between the two variables at the endpoints of the arrow in RSI, RCI, or
RU I (here, X and Y ), the reversible pieces of information are still included in the
same type of PID atom (redundancy, synergy, or unique information with respect to the
third variable). For example, the orange block in Fig. 3 indicates a common amount of
Z

redundancy in both PID diagrams: as such, RSI(X ↔ Y ) contributes both to redundant
information that Y and Z share about X, and to redundant information that X and
Z share about Y . By construction, these reversible components are symmetric in the
reversed variables. Note that, when we reverse the role of two variables, the third
variable (here, Z) remains a source and is thus put in the middle of our notation in
Z

Eqs. 3a, 3b and 3c. We also define the Irreversible Shared Information IRSI(X ← Y )
between X and Y considering Z (the light blue block in Fig. 3) as follows:
Z

Z

IRSI(X ← Y ) := SI(X : {Y ; Z}) − RSI(X ↔ Y ).

(4)

The attribute irreversible in the above definition indicates that this piece of redundancy
is specific to one of the two PIDs alone. More precisely, the uni-directional arrow in
Z
IRSI(X ← Y ) indicates that this piece of information is a part of the redundancy with
X as a target, but it is not a part of the redundancy with Y as a target2 . CorrespondZ
Z
ingly, at least one between IRSI(X ← Y ) and IRSI(Y ← X) is always zero. More
generally, IRSI quantifies asymmetries between two different PIDs: for example, when
Z
moving from the left to the right PID in Fig. 3, the light blue block IRSI(X ← Y ) indicates an equivalent amount of information that is lost for the redundancy SI(X : {Y ; Z})
and the synergy CI(X : {Y ; Z}) atoms, and is instead counted as a part of the unique
information U I(Y : {X\Z}) atom. In other words, assuming that the two redundancies
are ranked as in Fig. 3, we find that:
Z

IRSI(X ← Y ) = SI(X : {Y ; Z}) − SI(Y : {X; Z})

(5a)

= CI(X : {Y ; Z}) − CI(Y : {X; Z})

(5b)

= U I(Y : {X\Z}) − U I(X : {Y \Z}).

(5c)

While the coarser Shannon information quantities that are decomposed in both diagrams
in Fig. 3, namely I(X : Y ) and I(X : Y |Z), are symmetric under swap of X ↔ Y , their
2

In this paper, directional arrows never represent any kind of causal directionality: the PID framework
is only capable to quantify statistical (correlational) dependencies.

8

PID decompositions (see Eqs. 1 and 2) are not: Eq. 5 show that IRSI quantifies
the amount of this asymmetry. More precisely, the PID decompositions of I(X : Y )
Z

and I(X : Y |Z) will preserve the X ↔ Y symmetry if and only if IRSI(X ← Y ) =
Z

IRSI(Y ← X) = 0. Note that, in general, the differences of redundancies, of synergies,
and of unique information terms are always constrained by equations such as Eq. 5.
Hence, unlike for the reversible measures, we do not need to consider independent notions
of irreversible synergy or irreversible unique information.
In summary, the four subatoms in Eqs. 3a, 3b, 3c and 4, together with the two
remaining unique information terms (the black dots in Fig. 3), allow us to characterize
both PIDs in Fig. 3 and to understand how the PID atoms change when moving from
one PID to another. Note that in all cases the blocks indicate amounts of information,
but the interpretation of this information depends on the classification of variables as
target and sources within each diagram.

3.2

Unveiling the finer structure of the PID framework

So far we have examined the relationships among the PID atoms corresponding to two
different perspectives we hold about the system, whereby we reverse the roles of target
and source between two variables in the system. We have seen that the PID atoms of
different diagrams are not independent, as they are constrained by equations of the type
of Eqs. 1 and 2. More specifically, the eight PID atoms of two diagrams can be expressed
in terms of only six independent quantities, including reversible and irreversible pieces
of information. The next question is how many subatoms we need to describe all the
three possible PIDs (see Fig. 2). Since there are six constraints, three equations of the
type of Eq. 1 and three equations of the type of Eq. 2, one may be tempted to think
that the twelve PID atoms of all three PID diagrams can be expressed in terms of only
six independent quantities. However, the six constraints are not independent: this is
most easily seen from the symmetry of the co-information measure [31], which is defined
as the mutual information of two variables minus their conditional information given
the third (e.g., Eq. 1 minus Eq. 2). The co-information is invariant to any permutation
of the variables, and this property highlights that only five of the six constraints are
linearly independent. Accordingly, we will now detail how seven subatoms are sufficient
to describe the whole set of PIDs: we call these subatoms the minimal subatoms’ set of
the PID diagrams.
In Fig. 4 we see how the minimal subatoms’ set builds the PID diagrams. We assume,
without loss of generality, that SI(Y : {X; Z}) ≤ SI(X : {Y ; Z}) ≤ SI(Z : {X; Y }).
Then, we consider the three possible instances of Eq. 5 for the three possible choices of
the target variable, and we find that the same ordering also holds for the synergy atoms:
CI(Y : {X; Z}) ≤ CI(X : {Y ; Z}) ≤ CI(Z : {X; Y }). This property is related to the
invariance of the co-information: indeed, Ref. [10] indicated that the co-information can
be expressed as the difference between the redundancy and the synergy within each PID
diagram, i.e.
coI(X; Y ; Z) = SI(i : {j; k}) − CI(i : {j; k}),
for any assignment of X, Y , Z, to i, j, k.

9

(6)

a

Increasing

b

Figure 4: Constructing the full structure of the three PID diagrams in Fig. 2 in terms of a
minimal set of information subatoms. a: All the 12 PID atoms in Fig. 2 can be expressed
as sums of seven independent PID subatoms that are displayed as coloured blocks in
b (as in Fig. 2, the orientations of the PIDs for I(X : (Y, Z)) and I(Z : (X, Y )) are
rotated with respect to the PID for I(Y : (X, Z)) to highlight their reciprocal relations).
Five of these subatoms are reversible pieces of information, that are included in the
same kind of PID atom across different PID diagrams; the other two subatoms are the
irreversible pieces of information, that can be included in different kinds of PID atom
across different diagrams. Assuming, without loss of generality, that SI(Y : {X; Z}) ≤
Z

SI(X : {Y ; Z}) ≤ SI(Z : {X; Y }), the five reversible subatoms are: RSI(X ↔ Y )
Z

Z

X

(orange), RCI(X ↔ Y ) (gray), RU I(X ↔ Y ) (magenta), RU I(Y ↔ Z) (blue), and
Y

Z

RU I(X ↔ Z) (brown). The two irreversible subatoms are IRSI(X ← Y ) (light blue)
Y

and IRSI(Z ← X) (yellow).
These ordering relations are enough to understand the nature of the minimal subatoms’ set: we start with the construction of the three redundancies, which can all be expressed in terms of the smallest RSI and two subsequent increments. In Fig. 4, these corZ
Z
respond respectively to RSI(X ↔ Y ) = SI(Y : {X; Z}) (orange block), IRSI(X ← Y )

10

Y

(light blue block) and IRSI(Z ← X) (yellow block). In parallel, we can construct the
three synergies with the smallest RCI and the same increments used for the redundanZ
cies. In Fig. 4, these correspond respectively to RCI(X ↔ Y ) = CI(Y : {X; Z})
(gray block) and the same two IRSI used before. To construct the unique information
atoms, it is sufficient to further consider the three independent RU I defined by taking
all possible permutations of X, Y and Z in Eq. 3c. In Fig. 4, these correspond to
Z
Y
RU I(X ↔ Y ) = U I(X : {Y \Z}) (magenta block), RU I(X ↔ Z) = U I(Z : {X\Y })
X

(brown block), and RU I(Y ↔ Z) = U I(Z : {Y \X}) (blue block). We thus see that,
in total, seven minimal subatoms are enough to build the three PID diagrams of any
system. Among these seven building blocks, five are reversible pieces of information,
i.e. they contribute to the the same kind of PID atom across different PID diagrams;
the other two are irreversible pieces of information, that contribute to different kinds of
PID atom across different diagrams. The complete minimal set can only be determined
when all three PIDs are jointly considered and compared: as shown in Fig.3, pairwise
PIDs’ comparisons can at most distinguish two subatoms in any redundancy (or synergy), while the three-wise PIDs’ comparison discussed above allowed us to discern three
subatoms in SI(Z : {X; Y }) (and CI(Z : {X; Y }); see also Fig.4).
Importantly, while the definition of a single PID lattice relies on the specific perspective adopted on the system, which labels two variables as the sources and one variable
as the target, the decomposition in Fig. 4 is invariant with respect to the classification
of the variables. As described above, it only relies on computing all three PID diagrams
and then using the ordering relations of the atoms, without any need to classify the variables a priori. As illustrated in Fig. 4, the decomposition of the mutual information and
conditional mutual information quantities in terms of the subatoms is also independent
of the PID adopted. Our invariant minimal set in Fig. 4 thus extends the descriptive
power of the PID framework beyond the limitations that were intrinsic to considering
an individual PID diagram. In the next sections, we will show how the invariant minimal set can be used to identify the part of the redundant information about a target
that specifically arises from the mutual information between the sources (the source
redundancy), and to decompose the total entropy of any trivariate system.
Remarkably, the decomposition in Fig. 4 does not rely on any extension of Williams
and Beer’s axioms. We unveiled finer structure underlying the PID lattices just by
considering more PID lattices at a time and comparing PID atoms across different
lattices. We further remark that the decomposition in Fig. 4 does not rely in any
respect on the specific definition of the PID measures that is used to calculate the PID
atoms: it only relies on the axiomatic PID construction presented in Ref. [10].

4

Quantifying source redundancy

The structure of the three PID diagrams that was unveiled with the construction in
Fig. 4 enables a finer characterization of the modes of information distribution among
three variables than what has previously been possible. In particular, we will now
address the open problem of quantifying the source redundancy, i.e. the part of the
redundancy that ’must already manifest itself in the mutual information between the
sources’ [22]. Consider for example the redundancy SI(X : {Y ; Z}) in Fig. 4: it is
11

Z

Z

composed by RSI(X ↔ Y ) (orange block) and IRSI(X ← Y ) (light blue block).
We can check which of these subatoms are shared with the mutual information of the
sources I(Y : Z). To do this, we have to move from the middle PID diagram in Fig. 4,
that contains SI(X : {Y ; Z}), to any of the other two diagrams, that both contain
I(Y : Z). Consistently, in these other two diagrams I(Y : Z) is composed by the same
four subatoms (the orange, the light blue, the yellow and the blue block), and the only
difference across diagrams is that these subatoms are differently distributed between
unique information and redundancy PID atoms. In particular, we can see that both
the orange and the light blue block which make up SI(X : {Y ; Z}) are contained in
I(Y : Z). Thus, whenever any of them is nonzero, we know at the same time that Y
and Z share some information about X (i.e., SI(X : {Y ; Z}) > 0) and that there are
correlations between Y and Z (i.e., I(Y : Z) > 0). Accordingly, in the scenario of Fig. 4
the entire redundancy SI(X : {Y ; Z}) is explained by the mutual information of the
sources: all the redundant information that Y and Z share about X arises from the
correlations between Y and Z.
If we then consider the redundancy SI(Y : {X; Z}), that coincides with the orange
block, we also find that it is totally explained in terms of the mutual information between
the corresponding sources I(X : Z), which indeed contains an orange block. However,
if we consider the third redundancy SI(Z : {X; Y }), that is composed by an orange,
a light blue and a yellow block, we find that only the orange and the light blue block
Y
contribute to I(X : Y ), while the yellow does not. This means that if IRSI(Z ← X) > 0
(yellow block), then X and Y can share information about Z (i.e., SI(Z : {X; Y }) > 0)
even if there is no mutual information between the sources X and Y (i.e., I(X : Y ) = 0).
Following this reasoning, we define the source redundancy that two sources S1 and S2
share about a target T as:
S

S

SR(T : {S1 ; S2 }) := max{RSI(T ↔2 S1 ) , RSI(T ↔1 S2 )}
= max{min [SI(T : {S1 ; S2 }), SI(S1 : {S2 ; T })] ,

(7)

min [SI(T : {S1 ; S2 }), SI(S2 : {S1 ; T })]}.
One can easily verify that Eq. 7 identifies the blocks that belong to both SI(T, {S1 , S2 })
and I(S1 : S2 ) in Fig. 4, for any choice of sources and target (for instance, T = Z, S1 =
X and S2 = Y ). This definition can be justified as follows: each RSI measure in Eq. 7
compares SI(T : {S1 ; S2 }) with one of the other two redundancies that are contained
in the mutual information between the sources I(S1 : S2 ) (namely, SI(S1 : {S2 ; T }) and
SI(S2 : {S1 ; T })). Some of the subatoms included in I(S1 : S2 ) are contained in one of
these two redundancies, but not in the other, as they move to the unique information
mode when we change PID. Therefore, by taking the maximum in Eq. 7 we ensure that
SR(T : {S1 ; S2 }) captures all the common subatoms of I(S1 : S2 ) and SI(T : {S1 ; S2 }).
In a complementary way, we can define the non-source redundancy that two sources
share about a target:
N SR(T : {S1 ; S2 }) := SI(T : {S1 ; S2 }) − SR(T : {S1 ; S2 }).

12

(8)

I(X:Z)
UI(Z: {X\Y})

X

SR(Z:{X;Y})

SI(Z: {X;Y})

I(X:Y)

Y
UI(Z: {Y\X})

NSR(Z:{X;Y})

Z
CI(Z: {X;Y})

I(Y:Z)

Figure 5: Exploded view of the information that two variables X, Y carry about a third
variable Z. The mutual informations of each source about the target are decomposed
into PID atoms as in Fig. 1b, but the PID atoms are now further decomposed in terms of
the minimal subatoms’ set: the thick colored lines represent the subatoms with the same
colour code as in Fig. 4. Here, we assume that the variables are ordered as in Fig. 4.
The finer structure of the PID atoms allows us to identify the source redundancy SR(Z :
{X; Y }) (orange line + light blue line) as the part of the full redundancy SI(Z : {X; Y })
that is apparent in the mutual information between the sources I(X : Y ). Instead, the
amount of information in the non-source redundancy N SR(Z : {X; Y }) (yellow line)
also appears in the synergy CI(Z : {X; Y }).

4.1

The difference between source and non-source redundancy

Eqs. 7 and 8 show how we can split the redundant information that two sources
share about a target into two nonnegative information components: when the sourceredundancy SR is larger than zero there are also correlations between the sources, while
the non-source redundancy N SR can be larger than zero even when the sources are
independent. SR is thus seen to quantify the pairwise correlations between the sources
that also produce redundant information about the target: this discussion is pictorially
summarized in Fig. 5. In particular, the source redundancy SR is clearly upper-bounded
by the mutual information between the sources, i.e.
SR(T : {S1 ; S2 }) ≤ I(S1 : S2 ).

(9)

On the other hand, N SR does not arise from the pairwise correlations between the
sources: let us calculate N SR in a paradigmatic example that was proposed in Ref. [22]
to remark the subtle possibility that two statistically independent variables can share
information about a third variable. Suppose that Y and Z are uniform binary random
variables, with Y ⊥
⊥ Z, and X is deterministically fixed by the relationship X = Y ∧
Z. Here, SI(X : {Y ; Z}) ≈ 0.311 bit (according to different measures of redundancy
[22, 11]) even if I(Y : Z) = 0. Indeed, from our definitions in Eqs. 7 and 8 we find
that, since SI(Y : {X; Z}), SI(Z : {X; Y }) ≤ I(Y : Z) = 0, here N SR(X : {Y ; Z}) =
SI(X : {Y ; Z}) > 0 even though I(Y : Z) = 0. We will comment more extensively on
this instructive example in Section 6.
13

Interestingly, the non-source redundancy N SR is a part of the redundancy that is
related to the synergy of the same PID diagram. Indeed, two of the three possible N SR
defined in Eq. 8 are always zero, and the third can be larger than zero if and only if the
yellow block in Fig. 4 is larger than zero. From Fig. 4 and Fig. 5 we can thus see that,
whenever we find positive non-source redundancy in a PID diagram, the same amount of
information (the yellow block) is also present in the synergy of that diagram. Thus, while
there is source redundancy if and only if there is mutual information between the sources,
the existence of non-source redundancy is a sufficient (though not necessary) condition
for the existence of synergy. We can thus interpret N SR as redundant information about
the target that implies that the sources carry synergistic information about the target:
we give a graphical characterization of N SR in Fig. 5.
In the specific examples considered in [22], where the underlying causal structure
of the system is such that the sources always generate the target, the non-source redundancy can indeed be associated with the notion of ’mechanistic redundancy’ that
was introduced in that work: the causal mechanisms connecting the target with the
sources induce a non-zero N SR that contributes to the redundancy independently of
the correlations between the sources. In general, since the causal structure of the analyzed system is unknown, it is impossible to quantify ’mechanistic redundancy’ with
statistical measures, while it is always possible to quantify and interpret the non-source
redundancy as described in Section 4.
In section 6 we will examine concrete examples to show how our definitions of source
and non-source redundancy refine the information-theoretic description of trivariate systems, as they quantify qualitatively different ways that two variables can share information about a third.
We conclude this Section with more general comments about our quantification of
source redundancy. We note that the arguments used to define the source redundancy
in Sec. 4 can be equally used to study common or exclusive information components of
other PID terms. For example, we can identify the magenta subatom as the component
of the mutual information between the sources X and Y that cannot be related to their
redundant information about Z. Similarly, we could consider which part of a synergy is
related to the conditional mutual information between the sources.

5

Decomposing the joint entropy of a trivariate system

Understanding how information is distributed in trivariate systems should also provide
a descriptive allotment of all parts of the joint entropy H(X, Y, Z) [21, 30]. For comparison, Shannon’s mutual information enables a semantic decomposition of the bivariate
entropy H(X, Y ) in terms of univariate conditional entropies and I(X : Y ), that quantifies shared fluctuations (or covariations) between the two variables [30]:
H(X, Y ) = I(X : Y ) + H(X|Y ) + H(Y |X).

(10)

However, in spite of recent efforts [30], a univocal descriptive decomposition of the
trivariate entropy H(X, Y, Z) is still missing to date. Since the PID axioms in Ref. [10]
decompose mutual information quantities, one might hope that the PID atoms could
also provide a descriptive entropy decomposition. Yet, at the beginning of Section 3, we
14

Y

X

X

X1 X 2

X1 X 2

Y2
Y1
x
0
0
1
1
2
2
3
3

Z1
Z2
y
0
2
0
2
1
3
1
3

z
0
1
2
3
0
1
2
3

Z

Y

p(x, y, z)
1/8
1/8
1/8
1/8
1/8
1/8
1/8
1/8

Y1
Y2
x
0
1
0
1
2
3
2
3

(a)

Z1
Z2
y
0
1
2
3
0
1
2
3

z
0
1
2
3
2
3
0
1

Z

p(x, y, z)
1/8
1/8
1/8
1/8
1/8
1/8
1/8
1/8

(b)

Figure 6: Dyadic and triadic statistical dependencies in a trivariate system, as defined
in Ref. [21]. The tables display the non-zero probability values p(x, y, z) as a function of
the possible outcomes of the three stochastic variables X, Y, Z with domain {0, 1, 2, 3}.
X ∼ (X1 , X2 ), Y ∼ (Y1 , Y2 ), Z ∼ (Z1 , Z2 ) (the symbol ∼ here means ’is distributed as’),
where X1 , X2 , Y1 , Y2 , Z1 , Z2 are binary uniform random variables. a): The underlying
rules that give rise to dyadic dependencies are X1 = Y2 , Y1 = Z2 , Z1 = X2 . b): The
underlying rules that give rise to triadic dependencies are X1 = Y1 ⊕ Z1 , X2 = Y2 = Z2 .
pointed out that a single PID lattice does not include the mutual information between
the sources and their conditional mutual information given the target: this suggests
that a single PID lattice cannot in general contain the full H(X, Y, Z). More concretely,
Ref. [21] has recently suggested precise examples of trivariate dependencies where a single
PID lattice cannot account for, and thus describe the parts of, the full H(X, Y, Z).
These examples compare the dyadic and triadic dependencies described in Fig. 6.
Both kinds of dependencies underlie common modes of information sharing among three
and more variables [21]. Ref. [21] remarked that the atoms of a single PID diagram are
indeed able to distinguish between dyadic and triadic dependencies, but that such atoms
only sum up to two of the three bits of the full H(X, Y, Z).
We will now show how the missing allotment of the third bit of entropy in those
systems is not due to intrinsic limitations of the PID axioms, but just to the limitations
of considering a single PID diagram at a time — the common practice in the literature
so far. More generally, we will be able to allot and describe the full entropy H(X, Y, Z)
of any trivariate system in terms of the novel finer structure unveiled in Section 3.

15

5.1

The finer structure of the entropy H(X, Y, Z)

The minimal subatoms’ set that we illustrated in Fig. 4 allowed us to decompose all
three PID lattices of a generic system. However, to fully describe the distribution of
information in trivariate systems, we also wish to find a generalization of Eq. 10 to
the trivariate case, i.e. to decompose the full trivariate entropy H(X, Y, Z) in terms of
univariate conditional entropies and PID quantities. With this goal in mind, we first
subtract from H(X, Y, Z) the terms which describe statistical fluctuations of only one
variable (conditioned on the other two). The sum of these terms was indicated as H(1)
in Ref. [30], and there quantified as
H(1) = H(X|Y, Z) + H(Y |X, Z) + H(Z|Y, X).

(11)

This subtraction is useful because H(1) is a part of the total entropy which does not
overlap with any of the 12 PID atoms in Fig. 2. The remaining entropy H(X, Y, Z)−H(1)
was defined as the dual total correlation in Ref. [32] and recently considered in Ref. [30]:
DT C ≡ H(X, Y, Z) − H(1) .

(12)

DT C quantifies joint statistical fluctuations of more than one variable in the system. A
simple calculation yields
DT C = I(X : Y |Z) + I(Y : Z|X) + I(X : Z|Y ) + coI(X; Y ; Z),

(13)

which is manifestly invariant under permutations of X, Y and Z, and shows that DT C
can be written as a sum of some of the 12 PID atoms. For example, expressing the coinformation as the difference I(X : Z)−I(X : Z|Y ), we can arbitrarily use the four atoms
from the left-most diagram in Fig. 2 to decompose the sum I(X : Y |Z) + I(X : Z) and
then add U I(Y : {Z\X})+CI(Y : {X; Z}) from the middle diagram to decompose I(Y :
Z|X). If we then plug this expression of DT C in Eq. 12, we achieve a decomposition of
the full entropy of the system in terms of H(1) and PID quantities:
H(X, Y, Z) = H(1) + SI(X : {Y ; Z}) + U I(X : {Y \Z}) + U I(X : {Z\Y })+
+ CI(X : {Y ; Z}) + U I(Y : {Z\X}) + CI(Y : {X; Z}),

(14)

which provides a nonnegative decomposition of the total entropy of any trivariate system.
However, this decomposition is not unique, since the co-information can be expressed in
terms of different pairs of conditional and unconditional mutual informations, according
to Eq. 6. This arbitrariness strongly limits the descriptive power of this kind of entropy
decompositions, because the PID atoms on the RHS can only be interpreted within
individual PID perspectives.
To address this issue, we construct a less arbitrary entropy decomposition by using
the invariant minimal subatoms’ set that was presented in Fig. 4: importantly, that set
can be interpreted without specifying an individual, and thus partial, PID point of view
that we hold about the system.
Thus, after we name the variables of the system such that SI(Y : {X; Z}) ≤ SI(X :
{Y ; Z}) ≤ SI(Z : {X; Y }), we express the coarser PID atoms in Eq. 14 in terms of the

16

Figure 7: The joint entropy of the system H(X, Y, Z) is decomposed in terms of the
minimal set identified in Fig. 4, once the univariate fluctuations quantified with H(1)
have been subtracted out (see Eq. 15). As in Fig. 4, we assume without loss of generality
SI(Y : {X; Z}) ≤ SI(Z : {X; Y }) ≤ SI(X : {Y ; Z}). The finer PID structure unveiled
in this work enables a general entropy decomposition in terms of quantities that can be
interpreted without relying on a specific PID point of view, even though they have been
defined within the PID framework. The colored areas represent the Shannon information
quantities that are included in the DT C of Eq. 13, with the same color code of Fig. 2.
minimal set to obtain:
Z

Z

H(X, Y, Z) − H(1) = RSI(Y ↔ X) + 2 RCI(Y ↔ X)+
Z

X

(15)
Y

+ RU I(X ↔ Y ) + RU I(Y ↔ Z) + RU I(X ↔ Z)+
Y

Z

+ 2 IRSI(Z ← X) + 3 IRSI(X ← Y ).
Unlike Eq. 14, the entropy decomposition expressed in Eq. 15 and illustrated in Fig. 7
fully describes the distribution of information in trivariate systems without the need of a
specific perspective about the system. Importantly, this decomposition is unique: even
though the co-information can be expressed in different ways in terms of conditional and
unconditional mutual informations, in terms of the subatoms it is uniquely represented
as the orange block minus the gray block (see Fig. 7). Similarly, the conditional mutual
information terms of Eq. 13 are composed by the same blocks independently of the PID,
as highlighted in Fig. 4.

5.2

Describing H(X, Y, Z) for dyadic and triadic systems

To test the usefulness of the finer entropy decomposition in Eq. 15, we now compute
its terms for the dyadic and the triadic dependencies considered in Ref. [21] and defined in Figure 6. In both cases H(1) = 0. For the dyadic system, there are only
three positive quantities in the minimal set: the three reversible unique informations
Y
X
Z
RU I(X ↔ Z) = RU I(Y ↔ Z) = RU I(X ↔ Y ) = 1 bit. For the triadic system, there
Z

Z

are only two positive quantities in the minimal set: RSI(X ↔ Y ) = RCI(X ↔ Y ) = 1
Z

bit, but RCI(X ↔ Y ) is counted twice in the DT C. We illustrate the resulting entropy
decompositions, according to Eq. 15, in Fig. 8.
The decompositions in Fig. 8 enable a clear interpretation of how information is
finely distributed within dyadic and triadic dependencies. The three bits of the total

17

Figure 8: The joint entropy H(X, Y, Z) = 3 bit of a dyadic (upper panel) and a triadic
(lower panel) system, as defined in Fig. 6, is decomposed in terms of the minimal set as
illustrated in Fig. 7. In the dyadic system, H(X, Y, Z) is decomposed into three pieces
of (reversible) unique information: each variable contains 1 bit of unique information
with respect to the second variable about the third variable. In the triadic system,
H(X, Y, Z) is decomposed into one bit of information shared among all three variables
(the reversible redundancies) and two bits of (reversible) synergistic information due to
the three-wise XOR structure.
H(X, Y, Z) in the dyadic system are seen to be distributed equally among unique information modes: each variable contains 1 bit of unique information with respect to the
second variable about the third variable. Further, these unique information terms are
all reversible, which reflects the symmetry of the system under pairwise swapping of the
variables. This description provides a simple and accurate summary of the total entropy
of the dyadic system, which matches the dependency structure illustrated in Fig. 6a.
The three bits of the total H(X, Y, Z) in the triadic system consist of one bit of
the smallest reversible redundancy and two bit of the smallest reversible synergy, since
the latter appears twice in H(X, Y, Z). Again, the reversible nature of these pieces
of information reflects the symmetry of the system under pairwise swapping of the
variables. Further, the bit of reversible redundancy represents the bit of information
that is redundantly available to all three variables, while the two bit of reversible synergy
are due to the three-wise XOR structure (see Fig. 6b). Why does the XOR structure
provide two bits of synergistic information? Because if X = Y ⊕Z then the only positive
quantity in the set of subatoms in Fig. 4b is the smallest RCI, which however appears
twice in the entropy H(X, Y, Z). Importantly, these two bits of synergy do not come
from the same PID diagram: our entropy decomposition in Eq. 15 could account for
both bits only because it fundamentally relies on cross-comparisons between different
PID diagrams, as illustrated in Fig. 4.

18

6

Applications of the finer structure of the PID framework

The aim of this Section is to show the additional insights that the finer structure of the
PID framework, unveiled in Section 3.2 and Fig. 4, can bring to the analysis of trivariate
systems. We examine paradigmatic examples of trivariate systems and calculate the
novel PID quantities of source and non-source redundancy that we described in Sec. 4.
Most of these examples have been considered in the literature [22, 24, 11, 23, 30] to
validate the definitions, or to suggest interpretations, of the PID atoms. We also discuss
how SR matches the notion of source redundancy introduced in Ref. [22] and discussed
in Ref. [20]. Finally, we suggest and motivate a practical interpretation of the reversible
redundancy subatom RSI.
Even though our definitions of the minimal subatoms’ set in Fig. 4 only rely on
Williams and Beer’s axioms, some of the examples below will require a specific definition
of the PID atoms that goes beyond those axioms. In those cases, our computations rely
on the definitions of PID that were proposed in Ref. [11], which are the most widely
accepted in the literature for trivariate systems. Where numerical computations of the
PID atoms are involved, they have been performed with a software package that will be
publicly released upon publication of the present work.

6.1
6.1.1

Computing source and non-source redundancy
Copying — the redundancy arises entirely from source correlations

Consider a system where Y and Z are random binary variables that are correlated
according to a control parameter λ [22]. For example, consider a uniform binary random
variable W that ’drives’ both Y and Z with the same strength λ. More precisely,
p(y|w) = λ/2 + (1 − λ)δyw and p(z|w) = λ/2 + (1 − λ)δzw [22]. This system is completed
by taking X = (Y, Z), i.e. a two-bit random variable that reproduces faithfully the joint
outcomes of the generating variables Y, Z.
We consider the inputs Y and Z as the PID sources and the output X as the PID
target, thus selecting the left-most PID diagram in Fig. 2. Fig. 9a shows our calculations
of the full redundancy SI(X : {Y ; Z}), the source redundancy SR(X : {Y ; Z}) and the
non-source redundancy N SR(X : {Y ; Z}), based on the definitions in Ref. [11]. The
parameter λ is varied between λ = 0, corresponding to Y = Z, and λ = 1, corresponding
to Y ⊥
⊥ Z. Since N SR(X : {Y ; Z}) = 0 for any 0 ≤ λ ≤ 1, we interpret that all the
redundancy SI(X : {Y ; Z}) arises from the correlations between Y and Z (which are
tuned with λ). This is indeed compatible with the discussion in Ref. [22], where the
authors argued that in the ’copying’ example the entire redundancy should be already
apparent in the sources.
6.1.2

AND gate: the redundancy is not entirely related to source correlations

Consider a system where the correlations between two binary random variables, the
inputs Y and Z, are described by the control parameter λ as in 6.1.1, but the output X
is determined by the AN D function as X = Y ∧ Z [22]. As the causal structure of the
system would suggest, we consider the inputs Y and Z as the PID sources and the output

19

1

1
0.8

0.6

bit

bit

0.8

SI(output:finputsg)
SR(output:finputsg)
NSR(output:finputsg)

0.6

0.4

0.4

0.2

0.2

0
0

0.5

0
0

1

SI(output:finputsg)
SR(output:finputsg)
NSR(output:finputsg)

0.5

6

1

6

(a)

(b)

Figure 9: The binary random variables Y and Z are uniformly distributed inputs that
determine the output X as X = (Y, Z) in panel a) and as X = Y ∧ Z in panel b).
Correlations between Y and Z decrease with increasing λ, from perfect correlation (λ =
0) to perfect independence (λ = 1). The full redundancy SI(X : {Y ; Z}), the source
redundancy SR(X : {Y ; Z}) and the non-source redundancy N SR(X : {Y ; Z}) of the
inputs about the output are plotted as a function of λ. a Since the output variable
X copies the inputs (Y, Z), all of SI(X : {Y ; Z}) can only come from the correlations
between the inputs, which is reflected in N SR(X : {Y ; Z}) being identically 0 for all
values of λ. b For all values of λ > 0 we find N SR(X : {Y ; Z}) > 0, i.e. there
is a part of the redundancy SI(X : {Y ; Z}) which does not arise from the correlations
between the inputs Y and Z. Accordingly, N SR(X : {Y ; Z}) also appears in the synergy
CI(X : {Y ; Z}).
X as the PID target, thus selecting the left-most PID diagram in Fig. 2. Fig. 9b shows
our calculations of the full redundancy SI(X : {Y ; Z}), the source redundancy SR(X :
{Y ; Z}) and the non-source redundancy N SR(X : {Y ; Z}), based on the definitions in
Ref. [11].
SR and N SR now show a non-trivial behavior as a function of the Y − Z correlation
parameter λ. If λ = 0 and thus Y = Z, the full redundancy is made up entirely of
source redundancy — trivially, both SI(X : {Y ; Z}) and SR(X : {Y ; Z}) equal the
mutual information I(Y : Z) = H(Y ) = 1 bit. When λ increases, the full redundancy
decreases monotonically to its minimum value of ≈ 0.311 bit for λ = 1 (when Y ⊥
⊥ Z).
Importantly, SR(X : {Y ; Z}) decreases monotonically as a function of λ to its minimum
value of zero bit when λ = 1: this behavior is indeed expected from a measure that
quantifies correlations between the sources that also produce redundant information
about the target (see Section 4). On the other hand, N SR(X : {Y ; Z}) > 0 for λ > 0,
and it increases as a function of λ. When λ = 1, i.e. when Y ⊥
⊥ Z, N SR corresponds to
the full redundancy. This is compatible with our description of non-source redundancy
(see Section 4) as redundancy that is not related to the source correlations; indeed,

20

2.5

1.5

2.5
2

bit

bit

2

SI(output:finputsg)
SR(output:finputsg)
NSR(output:finputsg)

1.5

1

1

0.5

0.5

0
0

0.5

0
0

1

SI(output:finputsg)
SR(output:finputsg)
NSR(output:finputsg)

0.5

6

1

6

(a)

(b)

Figure 10: Two dice with uniformly distributed outcomes (y, z) ranging from 1 to 6 are
the inputs fed to an output third variable X as follows: a) x = y + z; b) x = y + 6 z.
The parameter λ controls the correlations between Y and Z (from complete correlation
for λ = 0 to complete independence for λ = 1). In panel a) the inputs Y, Z are
symmetrically combined to determine the output X, while in panel b) this symmetry
is lost: accordingly, for any fixed λ the full redundancy SI(X : {Y ; Z}) is larger in a)
than in b). Since the value of α does not change the inputs’ correlations, the relative
contribution of the inputs’ correlations to the full redundancy SI(X : {Y ; Z}) increases
from a) to b). Indeed, in panel a) N SR(X : {Y ; Z}) > 0 for λ > 0, while in b)
SI(X : {Y ; Z}) = SR(X : {Y ; Z}) for any λ, i.e. the redundancy arises entirely from
inputs’ correlations.
N SR > 0 implies that the sources also carry synergistic information about the target
(here, due to the relationship X = Y ∧ Z). N SR thus also quantifies the notion of
mechanistic redundancy that was introduced in Ref. [22] with reference to this scenario.
6.1.3

Dice sum: tuning irreversible redundancy

Consider a system where Y and Z are two uniform random variables, each representing
the outcome of a die throw [22]. A parameter λ controls the correlations between the
two dice: p(y, z) = λ/36 + (1 − λ)/6 δyz . Thus, for λ = 0 the dice throws always match,
for λ = 1 the outcomes are completely independent. Further, the output X combines
each pair (y, z) of input outcomes as x = y + α z, where α ∈ {1, 2, 3, 4, 5, 6}. This
example was suggested in Ref. [22] specifically to point out the conceptual difficulties in
quantifying the interplay between the redundancy and the source correlations, that we
addressed with the identification of SR and N SR (see Section 4).
Harder et al. calculated redundancy by using their own proposed measure Ired , which
also abides by the PID axioms in Ref. [10] but is different than the measure SI(X :
{Y ; Z}) introduced in Ref. [11]. However, Ref. [11] also calculated SI(X : {Y ; Z}) for
this example: they showed that the differences between SI(X : {Y ; Z}) and Ired in this
21

example are only quantitative (and there is no difference at all for some values of α),
while the qualitative behaviour of both measures as a function of λ is very similar.
Fig. 10 shows our calculations of the full redundancy SI(X : {Y ; Z}), the source
redundancy SR(X : {Y ; Z}) and the non-source redundancy N SR(X : {Y ; Z}), based
on the definitions in Ref. [11]. We display the two ’extreme’ cases α = 1, α = 6 for
illustration.
With α = 6, X is isomorphic to the joint variable (Y, Z): for any λ, this implies
on one hand SI(X : {Y ; Z}) = SI((Y ; Z) : {Y ; Z}) = I(Y : Z), and on the other
U I(Y : {Z\X}) = 0 and thus SI(Y : {X; Z}) = I(Y : Z) (see Eq. 1). According to
Eq. 7, we thus find that the source redundancy SR(X : {Y ; Z}) saturates, for any λ,
the general inequality in Eq. 9: all correlations between the inputs Y, Z also produce
redundant information about X (see Section 4). Further, according to Eq. 8, we find
N SR(X : {Y ; Z}) = 0 for any λ (see Fig. 10b): we thus interpret that all the redundancy
SI(X : {Y ; Z}) arises from correlations between the inputs. Instead, if we fix λ and
decrease α, the two inputs Y, Z are more and more symmetrically combined in the
output X: with α = 1, the pieces of information respectively carried by each input about
the output overlap maximally. Correspondingly, the full redundancy SI(X : {Y ; Z})
increases [22]. However, keeping λ fixed does not change the inputs’ correlations.
Thus, we expect that the relative contribution of the inputs’ correlations to the full
redundancy SI(X : {Y ; Z}) should decrease proportionally. Indeed, in Fig. 10a we find
N SR(X : {Y ; Z}) > 0 for λ > 0, which signals that a part of SI(X : {Y ; Z}) is not
related to the inputs’ correlations.
We finally note that also in this paradigmatic example the splitting of the redundancy
into SR and N SR addresses the challenge of separating the two kinds of redundancy
outlined in Ref. [22].
6.1.4

Trivariate jointly Gaussian systems

Barrett considered in detail the application of the PID to trivariate jointly Gaussian
systems (X, Y, Z) in which the target is univariate [23]: he showed that several specific
proposals for calculating the PID atoms all converge, in this case, to the same following
measure of redundancy:
SI(X : {Y ; Z}) = min[I(X : Y ), I(X : Z)].

(16)

We note that Eq. 16 highlights the interesting property that, in trivariate Gaussian
systems with a univariate target, the redundancy is as large as it can be, since it saturates
the general inequalities SI(X : {Y ; Z}) ≤ I(X : Y ), I(X : Z).
Direct application of our definitions in Eqs. 7 and 8 to such systems yields:
SR(X : {Y ; Z}) = min[I(X : Y ), I(X : Z), I(Y : Z)],
N SR(X : {Y ; Z}) = min[I(X : Y ), I(X : Z)] − min[I(X : Y ), I(X : Z), I(Y : Z)].
(17)
Thus, we find that in these systems the source redundancy is also as large as it can
be, since it also saturates the general inequalities SR(X : {Y ; Z}) ≤ I(X : Y ), I(X :

22

Z), I(Y : Z) (which follow immediately from its definition in Eq. 7). Further, combining
Eq. 16 with Eq. 17 gives:
(
SI(X : {Y : Z}), if I(Y : Z) ≥ SI(X : {Y ; Z}),
SR(X : {Y ; Z}) =
(18)
I(Y : Z),
otherwise;
(
0,
if I(Y : Z) ≥ SI(X : {Y ; Z}),
N SR(X : {Y ; Z}) =
SI(X : {Y : Z}) − I(Y : Z), otherwise.
(19)
This identification of source redundancy, which quantifies pairwise correlations between
the sources that also produce redundant information about the target (see Section 4),
provides more insight about the distribution of information in Gaussian systems. Indeed, the property that source redundancy is maximal indicates that the correlations
between any pair of source variables (for example, {Y ; Z} as considered above) produces
as much redundant information as possible about the corresponding target (X as considered above). Accordingly, when I(Y : Z) < SI(X : {Y ; Z}), the redundancy also
includes some non-source redundancy N SR(X : {Y ; Z}) > 0 that implies the existence
of synergy, i.e. CI(X : {Y ; Z}) > 0.

6.2

RSI quantifies information between two variables that also passes
monotonically through the third

In this section we discuss a practical interpretation of the reversible redundancy subatom
Z
RSI defined in Eq.3a. We note that RSI(X ↔ Y ) appears both in SI(X : {Y ; Z}) and
in SI(Y : {X; Z}), i.e. it quantifies a common amount of information that Z shares with
each of the two other variables about the third variable. We shorten this description
Z
into the statement that RSI(X ↔ Y ) quantifies ’information between X and Y that
also passes through Z’. Reversible redundancy is further characterized by the following
Proposition:
(Z,Z’)

Z

Proposition 1. RSI(X ↔ Y ) ≥ RSI(X ↔ Y ).
Proof.

Both SI(X : {Y ; (Z, Z 0 )}) ≥ SI(X : {Y ; Z}) and SI(Y : {X; (Z, Z 0 )}) ≥
Z

SI(Y : {X; Z}) [3]. Thus, RSI(X ↔ Y ) = min[SI(X : {Y ; Z}), SI(Y : {X; Z})] ≤
(Z,Z’)

min[SI(X : {Y ; (Z, Z 0 )}), SI(Y : {X; (Z, Z 0 )})] = RSI(X ↔ Y ).
Indeed, the fact that RSI always increases whenever we expand the middle variable
corresponds to the increased capacity of the entropy of the middle variable to host
information between the endpoint variables. We discuss this interpretation of RSI by
examining several examples, where we can motivate a priori our expectations about this
novel mode of information sharing.
6.2.1

Markov chains

Consider the most generic Markov chain X → Z → Y , defined by the property that
p(x, y|z) = p(x|z)p(y|z), i.e. that X and Y are conditionally independent given Z.
23

Figure 11: The finer structure of the three PID diagrams, as defined in Fig. 4, in the case
of a generic Markov chain X → Z → Y . Three of the seven subatoms of the minimal
set described in Fig. 4 are forced to zero by the Markov property I(X : Y |Z) = 0.
The Markov structure allows us to formulate a clear a priori expectation about the
amount of information between the endpoints of the chain, X and Y , that also ’passes
through the middle variable Z’: this information should clearly equal I(X : Y ), because
whatever information is established between X and Y must pass through Z. Indeed,
the Markov property I(X : Y |Z) = 0 implies, as we see immediately from Fig. 2, that
SI(X : {Y ; Z}) = SI(Y : {X; Z}) = I(X : Y ). By virtue of Eqs. 3a and 4, in accordance
with our expectation, we find
Z

RSI(X ↔ Y ) = I(X : Y ),

(20)

which holds true independently of the marginal distributions of X, Y and Z. Notably,
Z
the symmetry of RSI(X ↔ Y ) under swap of the endpoint variables X ↔ Y is also
compatible with the property that X → Z → Y is a Markov chain if and only if
Y → Z → X is a Markov chain. In words, whatever information flows from X through
Z to Y equals the information flowing from Y through Z to X.
Z
More generally, the Markov property I(X : Y |Z) = 0 implies that RCI(Y ↔ X) =
Z

Z

IRSI(X ← Y ) = RU I(X ↔ Y ) = 0: thus, only four of the seven subatoms of the
minimal set in Fig. 4 can be larger than zero. The three PIDs of the system, decomposed
Y
with the minimal set, are shown in Fig. 11. In particular, we see that also RSI(X ↔
Z) = I(X : Y ), thus matching our expectation that in the Markov chain X → Z → Y
the information between X and Z that also passes through Y still equals I(X : Y ).
We finally note that none of the results regarding Markov chains depends on specific
definitions of the PID atoms: they were derived only on the basis of the PID axioms in
Ref. [10].

24

6.2.2

Two parallel communication channels

Consider five binary uniform random variables X1 , X2 , Y1 , Y2 , Z with three parameters
λ
λ
λ
0 ≤ λ1 , λ2 , λ3 ≤ 1 controlling the correlations X1 ↔1 Z ↔2 Y1 , X2 ↔3 Y2 (in the same
λ

way λ controls the Y ↔ Z correlations in 6.1.1). We consider the trivariate system
(X, Y, Z) with X = (X1 , X2 ) and Y = (Y1 , Y2 ) (see Fig. 12). We intuitively expect
that the information between X and Y that also passes through Z, in this case, should
equal I(X1 : Y1 ), which in general will be smaller than I(X : Y ) = I(X1 : Y1 ) + I(X2 :
Y2 ). Indeed, we computed the PID atoms with the definitions in Ref. [11] over a fine
Y

discretization of the (λ1 , λ2 , λ3 )-space and we always found that RSI(X ↔ Z) = I(X1 :
Y1 ).

X1
X2

λ1

RSI((X1 , X 2 )

λ2

Z
λ3
Z

Y1
Y2

(Y1 , Y2 )) = I(X1 : Y1 )

Figure 12: X = (X1 , X2 ) and Y = (Y1 , Y2 ) share information via two parallel channels:
one passes through Z, the other does not. The parameters λ1 , λ2 , λ3 control the correlations as depicted with the arrows. In agreement with the interpretation in Section 6.2,
Z
RSI(X ↔ Y ) here quantifies information between X and Y that also passes through
Z, which clearly equals I(X1 : Y1 ).

6.2.3

Other examples
Z

To further describe the interpretation of RSI(X ↔ Y ) as information between X and
Y that also passes through Z, we here reconsider the examples, among those discussed
before Section 6.2, where we can formulate intuitive expectations about this information
sharing mode. In the dyadic system described in Fig. 6a, our expectation is that there
should be no information between two variables that also passes through the other:
Z
indeed, RSI(X ↔ Y ) = 0 in this case. Instead, in the triadic system described in
Fig. 6b, we expect that the information between two variables that also passes through
the other should equal the information that is shared among all three variables, which
Z
amounts to 1 bit. Indeed, we find RSI(X ↔ Y ) = 1 bit. In the ’copying’ example in
Section 6.1.1 X = (Y, Z), thus we expect that I(Y : Z) corresponds to information
between Y, Z that also passes through X, but also to information between X and Z
Z
Y
that also passes through Y . Indeed, RSI(X ↔ Y ) = RSI(X ↔ Z) = I(Y : Z). We
remark that all the values of RSI in these examples depend on the specific definitions
of the PID atoms in Ref. [11].

25

7

Discussion

The Partial Information Decomposition (PID) pioneered by Williams and Beer has
provided an elegant construction to quantify, with information theory, different ways
two stochastic variables can carry information about a third variable [10]. In particular,
it has enabled consistent definitions of synergy, redundancy and unique information
components among three stochastic variables [11]. More generally, it has generated
considerable interest as it addresses the difficult yet practically important problem of
extending information theory beyond the classic bivariate measures of Shannon to fully
characterize multivariate dependencies.
However, the axiomatic PID construction, as originally formulated by Williams and
Beer, fundamentally relied on the possibly arbitrary classification of the three variables
into sources and target, and this partial perspective prevented a complete and general
information-theoretic description of trivariate systems. More specifically, the original
PID framework could not quantify some important modes of information sharing, such
as source redundancy [22], and could not allot the full joint entropy of some important
trivariate systems [21]. The work presented here addresses these issues by extending the
original PID framework in two respects. First, we decomposed the original PID atoms in
terms of finer information subatoms with a well defined interpretation that is invariant
to different variables’ classifications. Then, we constructed an extended framework to
completely decompose the distribution of information within any trivariate system.
Importantly, our formulation did not require the addition of further axioms to the
original PID construction. We proposed that distinct PIDs for the same system, corresponding to different target selections, should be evaluated and then compared to
identify how the decomposition of information changes across different perspectives.
More specifically, we identified reversible pieces of information (RSI, RU I, RCI) that
contribute to the same kind of PID atom if we reverse the roles of target and source between two variables. The complementary subatomic components of the PID lattices are
the irreversible pieces of information (IRSI), that contribute to different kinds of PID
atom for different target selections. These subatoms thus measure asymmetries between
different decompositions of Shannon quantities pertaining to two different PIDs of the
same system, and such asymmetries reveal the additional detail with which the PID
atoms assess trivariate dependencies as compared to the coarser and more symmetric
Shannon information quantities.
The crucial result of this approach was unveiling the finer structure underlying the
PID lattices: we showed that an invariant minimal set of seven information subatoms is
sufficient to decompose the three PIDs of any trivariate system. In the remainder of this
section, possible uses of these subatoms and their implications for the understanding of
systems of three variables are discussed.

7.1

Use of the subatoms to map the distribution of information in
trivariate systems

Our minimal subatoms’ set was first used to characterize more finely the distribution
of information among three variables. We clarified the interplay between the redundant
information shared between two variables A and B about a third variable C, on one side,

26

and the correlations between A and B, on the other. We decomposed the redundancy
into the sum of source-redundancy SR, which quantifies the part of the redundancy
which arises from the pairwise correlations between the sources A and B, and nonsource redundancy N SR, which can be larger than zero even if the sources A and B
are statistically independent. Interestingly, we found that N SR quantifies the part of
the redundancy which implies that A and B also carry synergistic information about C.
The separation of these qualitatively different components of redundancy promises to
be useful in the analysis of any complex system where several inputs are combined to
produce an output [20].
Then, we used our minimal subatoms’ set to extend the descriptive power of the PID
framework in the analysis of any trivariate system. We constructed a general, unique,
and nonnegative decomposition of the joint entropy H(X, Y, Z) in terms of informationtheoretic components that can be clearly interpreted without arbitrary variable classifications. This construction parallels the decomposition of the bivariate entropy H(X, Y )
in terms of Shannon’s mutual information I(X : Y ). We demonstrated the descriptive power of this approach by decomposing the complex distribution of information in
dyadic and triadic systems, which was shown not to be possible within the original PID
framework [21].
We gave practical examples of how the finer structure underlying the PID atoms
provides more insight into the distribution of information within important and wellstudied trivariate systems. In this spirit, we put forward a practical interpretation of
the reversible redundancy RSI, and future work will address additional interpretations
of the components of the minimal subatoms’ set.

7.2

Possible extensions of the formalism to multivariate systems with
many sources

The insights that derive from our extension of the PID framework suggest that the PID
lattices could also be useful to characterize the statistical dependencies in multivariate
systems with more than two sources. Our approach does not rely on the adoption of
specific PID measures, but only on the axiomatic construction of the PID lattice. Thus,
it can be immediately extended to the multivariate case by embedding trivariate lattices
within larger systems’ lattices [12]: a further breakdown of the minimal subatoms’ set
could be obtained if the current system were embedded as part of a bigger system. Further, when systems with more than two sources are considered, the definition of source
redundancy might be extended as to determine which subatoms of a redundancy can
be explained by dependencies among the sources (for example, by replacing the mutual
information between two sources with a measure of the overall dependencies among all
sources, such as the total correlation introduced in Ref. [33]). More generally, the idea
of comparing different PID diagrams that partially decompose the same information can
also be generalized to identify finer structure underlying higher-order PID lattices with
different numbers of variables. These identifications might also help addressing specific
questions about the distribution of information in complex multivariate systems.

27

7.3

Potential implications for systems biology and systems neuroscience

A common problem in system biology is to characterize how the function of the whole
biological system is shaped by the dependencies among its many constituent biological variables. In many cases, ranging from gene regulatory networks [34] to metabolic
pathways [14] and to systems neuroscience [35, 36, 37], an information-theoretic decomposition of how information is distributed and processed within different parts of the
system would allow a model-free characterization of these dependencies. The work discussed here can be used to shed more light on these issues by allowing to tease apart
qualitatively different modes of interaction, as a first necessary step to understanding
the causal structure of the observed phenomena [38].
In systems neuroscience, the decomposition introduced here may be important for
studying specific and timely questions about neural information processing. This work
can contribute to the study of neural population coding, that is the study of how the concerted activity of many neurons encodes information about ecologically relevant variables
such as sensory stimuli [39, 37]. In particular, a key characteristic of a neural population
code is the degree to which pairwise or higher-order cross-neuron statistical dependencies are used by the brain to encode and process information, in a potentially redundant
or synergistic way, across neurons [7, 8, 9] and across time [40]. Our work is also of
potential relevance to study another hot issue in neuroscience, that is the relevance of
the information about sensory variables carried by neural activity for perception [19, 41].
This is a crucial issue to resolve the diatribe about the nature of the neural code, that
is the set of symbols used by neurons to encode information and produce brain function [42, 43, 44]. Addressing this problem requires mapping the information in the
multivariate distribution of variables such as the stimuli presented to the subject, the
neural activity elicited by the presentation of such stimuli, and the behavioral reports
of the subject’s perception. More specifically, it requires characterizing the information
between the presented stimulus and the behavioral report of the perceived stimulus that
can be extracted from neural activity [19]. It is apparent that developing general decompositions of the information exchanged in multivariate systems, as we did here, is
key to succeeding in rigorously addressing these fundamental systems-level questions.

Acknowledgments
We are grateful to members of Panzeri’s Laboratory for useful feedback, and to P.
E. Latham, A. Brovelli and C. de Mulatier for useful discussions. This research was
supported by the Fondation Bertarelli.

Author contributions
All authors conceived the research; G.P., E.P. and D.C. performed the research; G.P.,
E.P. and D.C. wrote a first draft of the manuscript; all authors edited and approved the
final manuscript; S.P. supervised the research.

28

References
[1] C. E. Shannon. A mathematical theory of communication. Bell System Technical
Journal, 27(3):379–423, 1948.
[2] N. Ay, E. Olbrich, N. Bertschinger, and J. Jost. A unifying framework for complexity measures of finite systems. In Proc. European Conf. Complex Sys., volume 6,
2006.
[3] Nils Bertschinger, Johannes Rauh, Eckehard Olbrich, and Jürgen Jost. Shared
information – new insights and problems in decomposing information in complex
systems. In Proceedings of the ECCS 2012, Brussels, 2012.
[4] Giulio Tononi, Olaf Sporns, and Gerald M. Edelman. Measures of degeneracy
and redundancy in biological networks. Proceedings of the National Academy of
Sciences, 96(6):3257–3262, 1999.
[5] Mikhail Tikhonov, Shawn C. Little, and Thomas Gregor. Only accessible information is useful: insights from gradient-mediated patterning. Royal Society Open
Science, 2:150486, 2015.
[6] Nicholas Timme, Wesley Alford, Benjamin Flecker, and John M. Beggs. Synergy,
redundancy, and multivariate information measures: an experimentalist’s perspective. Journal of Computational Neuroscience, 36(2):119–140, 2014.
[7] G Pola, A Thiele, K P Hoffmann, and S Panzeri. An exact method to quantify the
information transmitted by different mechanisms of correlational coding. Network,
14:35–60, 2003.
[8] E. Schneidman, W. Bialek, and M. J. Berry. Synergy, redundancy, and independence in population codes. Journal of Neuroscience, 23(37):11539–11553, Dec 2003.
[9] P. E. Latham and S. Nirenberg. Synergy, redundancy, and independence in population codes, revisited. Journal of Neuroscience, 25(21):5195–5206, May 2005.
[10] Paul Williams and Randall Beer. Nonnegative decomposition of multivariate information. arXiv:1004.2515, 2010.
[11] Nils Bertschinger, Johannes Rauh, Eckehard Olbrich, Jürgen Jost, and Nihat Ay.
Quantifying unique information. Entropy, 16(4):2161–2183, 2014.
[12] Daniel Chicharro and Stefano Panzeri. Synergy and redundancy in dual decompositions of mutual information gain and information loss. Entropy, 19:71, 2017.
[13] Dimitris Anastassiou. Computational analysis of the synergy among multiple interacting genes. Molecular Systems Biology, 3(1):83, 2007.
[14] N. Lüdtke, S. Panzeri, M. Brown, D. Broomhead, J. Knowles, M.A. Montemurro,
and D.B. Kell. Information-theoretic sensitivity analysis: a general method for
credit assignment in complex networks. Journal of The Royal Society Interface,
5(19):223–235, 2008.
29

[15] John Watkinson, Kuo-Ching Liang, Xiadong Wang, Tian Zheng, and Dimitris
Anastassiou. Inference of regulatory gene interactions from expression data using three-way mutual information. Annals of the New York Academy of Sciences,
1158(1):302–313, 2009.
[16] L. Faes, D. Marinazzo, G. Nollo, and A. Porta. An information-theoretic framework
to map the spatiotemporal dynamics of the scalp electroencephalogram. IEEE
Transactions on Biomedical Engineering, 63(12):2488–2496, 2016.
[17] Xaq Pitkow, Sheng Liu, Dora E. Angelaki, Giuseppe C. DeAngelis, and Alexandre
Pouget. How can single sensory neurons predict behavior? Neuron, 87(2):411–423,
2015.
[18] R.M. Haefner, S. Gerwinn, J.H. Macke, and M. Bethge. Inferring decoding strategies from choice probabilities in the presence of correlated variability. Nature Neuroscience, 16:235–242, 2013.
[19] S. Panzeri, C. D. Harvey, E. Piasini, P. E. Latham, and T. Fellin. Cracking the
Neural Code for Sensory Perception by Combining Statistics, Intervention, and
Behavior. Neuron, 93(3):491–507, Feb 2017.
[20] Michael Wibral, Viola Priesemann, Jim W. Kay, Joseph T. Lizier, and William A.
Phillips. Partial information decomposition as a unified approach to the specification of neural goal functions. Brain and Cognition, 112:25 – 38, 2017.
[21] Ryan G James and James P Crutchfield. Multivariate dependence beyond shannon
information. arXiv:1609.01233v2, 2016.
[22] Malte Harder, Christoph Salge, and Daniel Polani. Bivariate measure of redundant
information. Physical Review E, 87(1):012130, 2013.
[23] Adam Barrett. Exploration of synergistic and redundant information sharing in
static and dynamical gaussian systems. Physical Review E, 91(5):052802, 2015.
[24] Virgil Griffith and Christof Koch. Quantifying synergistic mutual information. In
Guided Self-Organization: Inception, pages 159–190. Springer Berlin Heidelberg,
2014.
[25] Virgil Griffith, Edwin Chong, Ryan James, Christopher Ellison, and James Crutchfield. Intersection information based on common randomness. Entropy, 16(4):1985–
2000, 2014.
[26] Pradeep Kr. Banerjee and Virgil Griffith. Synergy, redundancy and common information. arXiv:1509.03706, 2015.
[27] Robin AA Ince. The partial entropy decomposition: Decomposing multivariate entropy and mutual information via pointwise common surprisal. arXiv:1702.01591v2,
2017.
[28] Johannes Rauh, Pradeep Banerjee, Eckehard Olbrich, Jürgen Jost, and Nils
Bertschinger. On extractable shared information. arXiv:1701.07805, 2017.
30

[29] S. Stramaglia, L. Angelini, G. Wu, J. M. Cortes, L. Faes, and D. Marinazzo. Synergetic and redundant information flow detected by unnormalized granger causality:
Application to resting state fmri. IEEE Transactions on Biomedical Engineering,
63(12):2518–2524, 2016.
[30] Fernando Rosas, Vasilis Ntranos, Christopher J. Ellison, Sofie Pollin, and Marian
Verhelst. Understanding interdependency through complex information sharing.
Entropy, 18:38, 2016.
[31] William J. McGill.
19(2):97–116, 1954.

Multivariate information transmission.

Psychometrika,

[32] Te Sun Han. Nonnegative entropy measures of multivariate symmetric correlations.
Information and Control, 36(2):133 – 156, 1978.
[33] S. Watanabe. Information theoretical analysis of multivariate correlation. IBM
Journal of Research and Development, 4(1):66–82, Jan 1960.
[34] Adam A. Margolin, Ilya Nemenman, Katia Basso, Chris Wiggins, Gustavo
Stolovitzky, Riccardo Dalla Favera, and Andrea Califano. Aracne: An algorithm
for the reconstruction of gene regulatory networks in a mammalian cellular context.
BMC Bioinformatics, 7, 2006.
[35] Bruno B. Averbeck, Peter E. Latham, and Alexandre Pouget. Neural correlations,
population coding and computation. Nat Rev Neurosci, 7, 2006.
[36] Rodrigo Quian Quiroga and Stefano Panzeri. Extracting information from neuronal
populations: information theory and decoding approaches. Nat Rev Neurosci, 10,
2009.
[37] Stefano Panzeri, Jakob H. Macke, Joachim Gross, and Christoph Kayser. Neural
population coding: combining insights from microscopic and mass signals. 19, 2015.
[38] Judea Pearl. Causality: Models, Reasoning and Inference. Cambridge University
Press, New York, NY, USA, 2nd edition, 2009.
[39] M. Shamir. Emerging principles of population coding: in search for the neural code.
Current Opinion in Neurobiology, 25.
[40] C. A. Runyan, E. Piasini, S. Panzeri, and C. D. Harvey. Distinct timescales of
population coding across cortex. Nature, in press, 2017.
[41] Mehrdad Jazayeri and Arash Afraz. Navigating the neural space in search of the
neural code. Neuron, 93, 2017.
[42] Juan A. Gallego, Matthew G. Perich, Lee E. Miller, and Sara A. Solla. Neural
manifolds for the control of movement. Neuron, 94(5):978–984, 2017.
[43] Tatyana O. Sharpee. Optimizing neural information capacity through discretization. Neuron, 94(5):954–960, 2017.
[44] Xaq Pitkow and Dora E. Angelaki. Inference in the brain: Statistics flowing in
redundant population codes. Neuron, 94(5):943–953, 2017.

31

