Explainable Artificial Intelligence in Digital Pathology

arXiv:1712.06657v1 [] 18 Dec 2017

Towards the Augmented Pathologist:
Challenges of Explainable-AI in Digital Pathology
Andreas Holzinger1
Bernd Malle1,2
Peter Kieseberg1,2,3
Peter M. Roth4
Heimo Müller1,5
Robert Reihs1,5
Kurt Zatloukal5
1
2
3
4
5

a.holzinger@hci-kdd.org
b.malle@hci-kdd.org
p.kieseberg@sba-research.org
pmroth@icg.tugraz.at
heimo.mueller@medunigraz.at
robert.reihs@medunigraz.at
kurt.zatloukal@medunigraz.at

Institute for Medical Informatics, Statistics & Documentation, Medical University Graz, Austria
Secure Business Austria, SBA Research gGmbH, Vienna, Austria
University of Applied Science St. Pölten, Austria
Institute of Computer Graphics and Vision, Graz University of Technology, Austria
Institute of Pathology, Medical University Graz, Austria

Abstract
Digital pathology is not only one of the most promising fields of diagnostic medicine, but
at the same time a hot topic for fundamental research. Digital pathology is not just the
transfer of histopathological slides into digital representations. The combination of different data sources (images, patient records, and *omics data) together with current advances
in artificial intelligence/machine learning enable to make novel information accessible and
quantifiable to a human expert, which is not yet available and not exploited in current
medical settings. The grand goal is to reach a level of usable intelligence to understand the
data in the context of an application task, thereby making machine decisions transparent,
interpretable and explainable. The foundation of such an ”augmented pathologist” needs
an integrated approach: While machine learning algorithms require many thousands of
training examples, a human expert is often confronted with only a few data points. Interestingly, humans can learn from such few examples and are able to instantly interpret
complex patterns. Consequently, the grand goal is to combine the possibilities of artificial
intelligence with human intelligence and to find a well-suited balance between them to enable what neither of them could do on their own. This can raise the quality of education,
diagnosis, prognosis and prediction of cancer and other diseases. In this paper we describe
some (incomplete) research issues which we believe should be addressed in an integrated
and concerted effort for paving the way towards the augmented pathologist.

1. Introduction
Artificial intelligence (AI) is currently experiencing a tremendous hype towards building
systems that can learn and think like humans do (Lake et al., 2016; Weber et al., 2017).
Much of the current success results from the applicability of deep neural networks trained
on large data sets in various tasks. These include object recognition, e.g. identifying cats
from unlabeled images (Le et al., 2011), video games, e.g. playing Atari games (Mnih et al.,
2013), or board games, e.g. the game of Go (Silver et al., 2016). These systems achieve a
performance that equals or even beats humans in some respects (Silver et al., 2017).
1

Holzinger, Malle, Kieseberg, Roth, Müller, Reihs, Zatloukal

An impressive example from the medical domain is the very recent work by the ThrunGroup (Esteva et al., 2017), which showed that convolutional neural networks can achieve a
performance on par with human doctors, demonstrating the ability of classifying skin cancer
with a level of competence comparable to dermatologists. Despite all these impressive successes of automatic approaches we are still very far away from reaching general human-level
AI. This would require algorithms to deal with the common sense ”informatic” situation we say context - in which the phenomena to be taken into account in achieving goals are
not yet fixed in advance (McCarthy, 2007).
In our opinion, there is no danger that AI will replace medical professionals in the
medium term; however, there is an increasing trend that those who will not enrich their
methods with AI will have a potential disadvantage to those who embrace cutting-edge
Machine Learning (ML) technologies.
AI and ML are often used synonymously, although there is a difference: AI is the field
working on understanding intelligence and encompasses all underlying scientific theories of
human intelligence/human learning versus machine intelligence/machine learning. ML is a
very practical field and applies the findings of AI research to the design, development and
testing of algorithms that can learn from data, gain knowledge from experience and improve
their learning behavior over time (Holzinger, 2017). Understanding human intelligence,
i.e. to study how humans make decisions and how they reason, is difficult but of utmost
importance for advances in AI and ML. For a number of reasons it is very difficult to deal
with non-stationary, non-linear as well as non-independent & non-identically distributed
data 1 in high-dimensional spaces, often additionally dependent on time. This is quite often
the case in the medical domain where we are confronted with a variety of different data (for
a taxonomy of data refer to (Holzinger et al., 2014d; Holzinger, 2014a)).
Studying the work of pathologists is interesting for several reasons: 1) Digital pathology
is not just the transformation of the classical microscopic analysis of histological slides by
pathologists to a digital visualization, it is an innovation that will dramatically change
medical workflows in the coming years; 2) Much information is hidden in arbitrarily high
dimensional spaces, not accessible to a human, consequently we need an AI/ML approach to
generate a new kind of information, which is not yet available and not exploited in current
diagnostics; 3) Pathologists are able to transfer previously learned knowledge quickly to new
tasks. Insights into the latter supports AI research generally and ML research specifically
and may contribute to developing software which can learn from experience, similarly as
we humans do. Technologically, during the last decade pathology has benefited from the
rapid progress of image digitizing technologies, which led to the development of scanners
capable to produce Whole Slide images (WSI) which can be explored by a pathologist on a
computer screen (virtual microscope) comparable to the conventional microscope and can be
used for education and training, diagnostics (clinico-pathological meetings, consultations,
revisions, slide panels and upfront clinical diagnostics) and archiving (Al-Janabi et al.,
2012). Practical implications of digital pathology include telepathology (Afework et al.,
1998), second opinions (Epstein et al., 1996) and education (Dee, 2009).

1. Learning from non-iid data is difficult, (Zhang et al., 2009)

2

Explainable Artificial Intelligence in Digital Pathology

The digitalization opens new opportunities to extract knowledge from such data, which
would not be accessible to the human expert otherwise (Holzinger et al., 2014a); however,
standard image analysis tools are inadequate to deal with the specialities of WSI’s.
Fusing traditional image sources with *omics data calls for new solutions (Madabhushi
and Lee, 2016), particularly if we want to go towards personalized medicine (Holzinger,
2014c). The variety of problems in digital pathology requires a synergistic combination of
various methodological approaches which calls for an integrated ML approach (Holzinger,
2012, 2013; Holzinger et al., 2017a). Methodologies of two areas offer ideal conditions to
implement integrated ML: Human–Computer Interaction (HCI) and Knowledge Discovery/Data Mining (KDD), with the goal of augmenting human intelligence with artificial
intelligence to discover novel, previously unknown insights into data. This calls for solutions which let the medical professional not only ask questions like ”Where are similarities/differences/anomalies ...?” but to ask ”why” questions, for example ”Why are there
similarities/differences/anomalies ...?” in order to find and explain unknown unknowns in
complex data, which goes beyond data mining. Interestingly, early developments from the
field of augmenting human intelligence came from HCI-research (Engelbart and English,
1968; Engelbart, 1995; Russell and Norvig, 1995).

2. From Microscopical Thinking to Digital Pathology
Modern pathology was founded in the mid of the 19th century, when Virchow (1871) set the
basis for modern medical science and established the ”microscopical thinking” which is still
state-of-the-art in classical pathology. In histopathology a biopsy or surgical specimen is
examined by a pathologist, after the specimen has been processed and histological sections
have been placed onto glass slides, see Figure 1. In cytopathology either free cells (fluids) or
tissue micro-fragments are ”smeared” on a slide without cutting any tissue. An alternative
to invasive autopsy is virtual autopsy which is also conducted with scanning and imaging
techniques (O’Sullivan et al., 2017).

Figure 1: Clinical pathology, from mid 19th century up to the end of the 20th century.
3

Holzinger, Malle, Kieseberg, Roth, Müller, Reihs, Zatloukal

Towards the end of the 20th century an individual clinical pathologist was no longer
able to cover the knowledge of the whole scientific field. This led to the specialization of
clinical pathology either by organ systems or methodologies. Molecular biology and *omics
technologies set the foundation for the emerging field of molecular pathology, which today
alongside WSI provides the most important source of information, especially in the diagnosis
of cancer and infectious diseases, see Figure 2.

Figure 2: Today’s state-of-the-art (non digital) clinical pathology.
The roots of digital pathology go back to the 1960s in the context of telepathology,
which was a remote-controlled light microscope attached to a video camera and a telecommunication link (Weinstein et al., 1987).
Later in the 1990s the principle of virtual microscopy (Weinstein et al., 2009) appeared
in several life science research areas. At the turn of the century the scientific community gradually adopted the term ”digital pathology” (Barbareschi et al., 2000) to denote
digitization efforts in pathology, see Figure 3.
The technical requirements (scanner, storage, network) in digital pathology are magnitudes higher than in radiology due to some general differences between those two fields: In
radiology the image is primarily captured in digital format, whilst in pathology the scanning is done from preserved and processed specimens, for retrospective studies even from
slides stored in biobanks (Huppertz and Holzinger, 2014). The integration of biobanks is of
enormous importance for the realization of future precision medicine (Müller et al., 2015).
4

Explainable Artificial Intelligence in Digital Pathology

Figure 3: A digital pathology workflow always starts with the gross evaluation of the primary sample. Depending on the medical question and the material type, small
tissue parts are extracted from the primary sample. The pathology labs cuts
several slides from the tissue blocks, applies different staining methods and conducts additional histological and molecular tests. In addition to the set of whole
slide images (each about 16 Gigapixels in size!), textual information from medical
reports as well as molecular data are combined in the diagnosis-making process.

Besides some differences in pre-analytics workflows and metadata content, the required
storage in digital pathology is two to three orders of magnitude higher than in radiology
and places heavy demands on IT infrastructure (Huisman et al., 2010).
2.1 Virtual Case
A pathological workflow always starts with the overall evaluation of the primary sample.
Depending on the medical question and the material type, small tissue parts are extracted
from the primary sample and are either embedded in a paraffin block or cryo-frozen. From
the tissue blocks the pathology lab cuts several slides, applies different staining methods
and conducts additional histological and molecular tests. Finally, the pathologists evaluate
all the slides together with the supporting gross- and molecular findings and makes the
diagnosis. If in addition to the set of WSI all information is present in a structured digital
format, we call this a virtual case. In a virtual case, the average number of slides and
additional findings varies very much for different medical questions and material types.
5

Holzinger, Malle, Kieseberg, Roth, Müller, Reihs, Zatloukal

The most demanding data elements in a virtual case are the whole slide images (WSI).
Compared to radiology, where the typical file sizes are in the range from 500 KB to 50 MB,
a single WSI scan with 80x magnification consists of approximately 16 Gigapixels (Note:
for the calculation of the WSI file size and comparison of different scanner manufacturers,
we use the de-facto standard area of 15mm x 15mm, with an optical resolution of 0.12 µm,
which corresponds to an 80x magnification).
With 8bit information for each color channel one WSI results in approx. 50GB stored
in an uncompressed image format. Looking at the number of slides of a typical case, it
is clear that some compression techniques must be applied to the image data, and luckily
several studies reported that lossy compression with a high quality level does not influence
the diagnostic results; Still there are unresolved questions (Holzinger et al., 2017a).
The newest generation of scanners (status as of October 20172 ) is able to digitize a slide
at various vertical focal planes, called z-layers, each the size of a singe layer. The multi-layer
image can be either combined by algorithms to a single composite multi-focus image (Zstacking) or used to simulate the fine focus control of a conventional microscope. Z-stacking
is a desirable feature especially when viewing cytology slides, however the pathologist should
be aware that such an image can never be seen through an actual microscope.
At the Institute of Pathology of the Medical University of Graz about 73,000 diagnoses
are made per year and approximately 335,000 glass slides are produced in the pathology
lab; additionally approximately 25,000 glass slides are generated in the cytology lab. This
results in a required yearly storage capacity of almost 1 PB and the appropriate computing
power to process approximately 1000 slides per day plus the necessary capacity to train and
improve AI/ML algorithms.
Several data formats are used today, either vendor independent (DICOM, TIFF/BigTIFF,
Deep Zoom images) and vendor specific formats from Aperio, Hamamatsu, Leica, 3DHistech, Philips, Sakura and Trestle. In the setup of a virtual slide archive for medical research
and machine learning it is essential to a) agree on a common exchange format, and b) to
separate patient related and image related metadata. Patient related metadata comprise
direct identifiers (name, birthday, zip code, ...) but also diagnosis results and others results
from the patient medical history. When no such data is stored within or attached to the
image format, the WSI is purely anonymous as no re-identification of the patient is possible.
To link between the same WSI used in different studies, either a global unique identifier
(GUID) or an image generated hash can be used.
2.2 Towards the Augmented Pathologist
Digitizing the workflows is one important fundamental issue to enable a revolutionary
change in clinical pathology. Based on the digital workflow AI paradigms and ML methods
can be applied to augment the human pathologist during diagnosing, research, education
and training. The augmentation may start with simple classification and quantification
algorithms as already available today (e.g. cell sorting (Arteta et al., 2012)), and may end
in a fully autonomous pathologist. This is similar to the concept of autonomous driving,
where human intervention is no longer necessary, but can still range over various degrees of
involvement (to-be-or-not-to-be-in-the-loop (Louw et al., 2015)).
2. As of October 2017 the worldwide fastest WSI scanner is working in Graz https://goo.gl/yiVnEx

6

Explainable Artificial Intelligence in Digital Pathology

In driving automation autonomy levels have already been established (SAE J3016 3 )
Kircher et al. (2014): level 0 = full control of the driver, just warnings; level 1 = hands-on,
driver must be ready to retake full control at any moment; level 2 = hands-off, driver must
monitor at any time and be prepared to immediately intervene in case the automatic system
fails; level 3 = eyes-off, driver can safely turn her/his attention elsewhere, but driver may
take over at any time; level 4 = mind-off, driver even may go to sleep or leave the driver’s
seat; level 5 = fully automatic, no human intervention required, e.g. a robotic taxi.
To distinguish such extreme scenarios from simple digital workflows we propose the term
machine aided pathology, meaning that significant contributions of the decision making
process are supported by machine intelligence. Such machine aided pathology solutions can
be applied at several steps during the diagnosis and decision making process:
Formulation of a hypothesis. Each diagnosis starts with a medical question and a corresponding underlying initial hypothesis. The pathologist refines this hypothesis in
an iterative process, consequently looking for known patterns in a systematic way
in order to confirm, extend or reject his/her initial hypothesis. Unconsciously, the
pathologist asks the question ”What is relevant?” and zooms purposefully into the according to his/her opinion - essential areas of the cuts. The duration and the error
rate in this step vary greatly between inexperienced and experienced pathologists. An
algorithmic support in this first step would contribute in particular to the quality and
interoperability of pathological diagnoses and reduce errors at this stage, and would be
particularly helpful for educational purposes. A useful approach is known from Reeder
and Felson (2003) to discover certain patterns (called gamuts) in radiological images
and to classify these into the groups ”common/uncommon”. This approach has its
roots in differential diagnosis which has a long tradition (Kobayashi et al., 1974) and
has been implemented with artificial neural networks (Asada and Doi, 1997).
- Very large amounts of data can only be managed with a ”multi resolution” approach
using image pyramids. For example, a Colon cancer case consists of approximately 20
Tera (!) pixels of data - a size which no human is capable of processing.
- The result of this complex process is a central hypothesis, which has to be tested on
a selection of relevant areas in the WSI, which is determined by quantifiable values
(receptor status, growth rate, etc.).
- Training data sets for ML can now contain human learning strategies (transfer
learning, e.g.) as well as quantitative results (hypotheses, areas, questions, etc.).
Detection and classification of known features. Through a precise classification and
quantification of selected areas in the sections, the central hypothesis is either clearly
confirmed or rejected. In this case, the pathologist has to consider that the entire
information of the sections is no longer taken into account, but only areas relevant
to the decision are involved. It is also quite possible that one goes back to the initial
hypothesis step by step and changes their strategy or consults another expert, if no
statement can be made on the basis of the classifications.
3. Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems.
Society of Automotive Engineers (SAE, 2014).

7

Holzinger, Malle, Kieseberg, Roth, Müller, Reihs, Zatloukal

- In this step ML algorithms consist of well known standard classification and quantification approaches.
- An open question is how to automatically or at least semi-automatically produce
training sets, because here specific annotations are needed (which could come from a
stochastic ontology, e.g.).
- Another very interesting and important research question is, whether and to what
extent solutions learned from one tissue type (organ 1) can be transferred to another
tissue type (organ 2) – transfer learning – and how robust the algorithms are with
respect to various pre-analytic methods, e.g. stainings, etc.
Risk prediction and identification of unknown features. Within the third step, recognized features (learned parameters) are combined into a diagnosis and an overall
prediction of survival risk. The main challenge in this step lies in training/validation
and in the identification of novel, previously unknown features from step two. We
hypothesize that the pathologist supported by machine learning approaches is able to
discover patterns – which previously were not accessible! This would lead to totally
new insights into previously unseen or unrecognized relationships.
Beyond challenges in ML, the following general topics and prerequisites have to be solved
for a successful introduction of machine aided pathology:
Standardization of WSI image formats and harmonization of annotation / metadata formats. This is essential for telepathology applications (on the importance of standardization for digital pathology see the recent work of Barisoni et al. (2017)); and even
more important for the generation of training sets, as for a specific organ and disease
stages, the required amount of cases may not be available even at a large institute of
pathology.
Common digital cockpit and visualization techniques should be used in education, training and across different institutes. Changing the workplace should be as easy as
switching the microscope model or manufacturer. However, commonly agreed-upon
visualization and interaction paradigms can only be achieved in a cross vendor approach and with the involvement of major professional associations.
2.3 Data Integration
Applying sophisticated AI/ML methods on images alone is one aspect (Liu et al., 2017);
however, the real added value results from the combination of two other sources of data:
1) Clinical data from electronic patient records (EPR) including the patient
history with all documents, diagnoses, medical reports, laboratory tests, physiological parameters, vital signs, but also recorded signals, ECG, EEG, etc.); this also links to other
image data including standard X-ray, MR, CT, PET, SPECT, microscopy, confocal laser
scans, ultrasound imaging, molecular imaging, etc.4
4. Remark: At Graz University Hospital an all-digital hospital information system is in operation since
2000 (Gell et al., 2000; Holzinger et al., 2000)

8

Explainable Artificial Intelligence in Digital Pathology

2)*omics data from biobanks (Huppertz and Holzinger, 2014; van Ommen et al.,
2015; Hainaut et al., 2017), e.g. from genomic sequencing technologies (Next Generation Sequencing, NGS, etc.), microarrays, transcriptomic technologies, proteomic and metabolomic
technologies, etc., which all play important roles for biomarker discovery and drug design
(McDermott et al., 2013; Swan et al., 2013; Libbrecht and Noble, 2015).
Fusing and integrating this data (images, text, *omics) allows to make novel information
accessible and quantifiable for the human pathologist, which are not yet available and not
exploited to date. To understand the whole one must study the whole (quote by Henrik
Kacser (1986), reference in: Kell (2004)). Data integration is a hot topic in the life sciences because solutions can bridge the gap between clinical routine and biomedical research
(Jeanquartier et al., 2016). This is becoming important due to the distributed, heterogeneous and diverse data sources, including picture archiving and communication systems
(PACS) and radiological information systems (RIS), hospital information systems (HIS), lab
information systems (LIS), physiological/clinical data repositories, and all sorts of *omics
data in labs, and biobanks. It is of highest importance to provide end-users with a unified
view on these data,. These unified views are particularly important in high-dimensions, e.g.
for integrating heterogeneous descriptions of the same set of genes (Lafon et al., 2006), and
fused data is more informative (Blanchet and Smolinska, 2016).

3. Deep Learning
Among the most important machine learning methods in medical image analysis are support
vector machines, random forests, and deep learning approaches. For a brief overview of
machine learning for digital pathology see (Holzinger et al., 2017a). Here we briefly focus
on the state-of-the-art in interpretable deep learning. Deep Learning (DL, for an overview
refer to (LeCun et al., 2015; Schmidhuber, 2015; Arel et al., 2010)) is a whole family of
(hierarchical) learning approaches primarily based on learning representations by layers
and units in a network structure. Due to their practical success DL is currently the most
popular general framework (Greenspan et al., 2016) in ML. The ”deeper” the network the
more layers and intra-layer units can represent functions of increasing complexity. Most
tasks that consist of mapping an input vector to an output vector can be accomplished via
deep learning, given sufficiently large data sets of labeled training examples (for details see
the excellent book by Goodfellow et al. (2016)). Despite all their successes DL approaches
have some limitations, e.g. they are very data hungry, need much computational power and
they are considered black-box models (Singh et al., 2016). However, in the medical domain
it is necessary to be able to open the black-box to a glass-box (Holzinger et al., 2017b) and
to make the results transparent, re-traceable and explainable on demand (see section 7).
3.1 Convolutional Neural Networks
The most prominent and recently most successful DL architectures are Convolutional Neural
Networks (CNN, or short: convnets), which have been introduced in the early 1990ies (LeCun et al., 1989), and which have demonstrated excellent performance recently (Krizhevsky
et al., 2012)5 . Technically a CNN is structured as a series of different layers, i.e. convolu5. This NIPS-paper has 17,770 citations as of 14.12.2017 12:00 CET

9

Holzinger, Malle, Kieseberg, Roth, Müller, Reihs, Zatloukal

Figure 4: Detail of a typical WSI: FFPE section of a human liver with cirrhosis, hematoxylin
and eosin staining (Image Source: Pathology Graz, obtained with the new P1000
scanner of 3DHistech), see also Figure 8

tional layers, pooling layers, fully connected layers and classification layers (e.g. softmax).
Convolutional layers are feature maps, where each feature map is connected to local patches
in the corresponding feature map in the previous layer. Pooling layers merge similar features
into one. Fully connected layers generate the output for the actual task and classification
layers produce a label provided a learning instance. CNNs are so popular because the effort
of engineering the feature representations can be mostly handled by the network itself.
10

Explainable Artificial Intelligence in Digital Pathology

Also in the medical domain, amazing results have been achieved, e.g. for cancer detection with human-like performance: Liu et al. (2017) adopted a CNN framework for breast
cancer metastasis detection in lymph nodes. By exploiting the information of a pre-trained
model, sophisticated image normalization, and building on a multi-stage approach (mimicking the human perception), state-of-the-art methods and even human pathologists have
been outperformed on a standard benchmark dataset.
Similarly, Esteva et al. (2017) addresses the problem of skin cancer detection: they
propose a transfer learning setup in which a pre-trained CNN architecture using ImageNet
has been modified in such a way that only the final classification layer is discarded and
re-trained for the given task (additionally, the parameters are fine-tuned across all layers).
Eventually, the obtained network was able to perform on par with human dermatologists
on different tasks.
Even though this demonstrates that DL may be beneficial in the medical domain generally, the main challenge is still to cope with the problem that often the rather large amount
of training data required is not available. Consequently, there has been a considerable interest in approaches that can learn from a small number of training samples (see section 6).
The most common and straight forward way is to use data augmentation for constructing
iterative optimization via the generation of artificial data or latent variables (Van Dyk and
Meng, 2001; Hauberg et al., 2016). In our domain image augmentation (Bloice et al., 2017)
can be particularly beneficial, where additional training samples are generated via variation
of the given data, e.g. via rotation, elastic deformation, adding noise, etc.
A prominent example for such an approach is U-Net (Ronneberger et al., 2015), which
demonstrated that state-of-the-art results can be obtained for bio-medical image segmentation even when the model was trained on merely a few samples.
Even though this simple approach often yields good results, it is restricted as only
limited variations can be generated from the given data. A promising direction is to build
on ideas from transfer learning (Long et al., 2016). The key idea is to pre-train a network
on large publicly available datasets and then to fine-tune it for the given task. For example,
Cai et al. (2017) fine-tunes the VGG-16 network, which is already pre-trained using a huge
amount of natural images, to finally segment pancreas from MR images. In addition, a
CRF step is added for the final segmentation. Another way would be to use specific prior
knowledge about the actual task (Payer et al., 2016).
However, this information is often not available and, as mentioned above, medical image
data and natural images are often not sharing the same characteristics, which is why such
approaches often fail in practice.
A totally different way to deal with small amounts of training data is to use synthetically
generated samples for training such as (Rozantsev et al., 2015) which are easy to obtain.
However, even in this way the specific characteristics of the given image data might not
be reflected properly. To overcome this problem, Generative Adversarial Nets (Goodfellow
et al., 2014) train a generator and a discriminator framework in parallel. The key idea is
that the generator synthesizes images and the discriminator decides if an image is real or
fake (i.e., generated by the generator). In this way, increasingly better training data can
be generated. This idea is for example exploited by Nie et al. (2017) to better model the
nonlinear relationship between CR and MR images.
11

Holzinger, Malle, Kieseberg, Roth, Müller, Reihs, Zatloukal

3.2 Interpretable Deep Learning Models
Simple models are regarded as more interpretable than complex ones, therefore linear models
and basic decision trees still dominate in many applications where interpretability is an issue.
This belief is however challenged by recent work, in which carefully designed interpretation
techniques have shed light on some of the most complex and deepest machine learning
models (Singh et al., 2016).
Standard supervised convnet models LeCun et al. (1989); Krizhevsky et al. (2012) map
a color 2D input image xi , via a series of layers, to a probability vector yˆi over C different
classes. Each layer consists of (i) a convolution of the previous layer output (or, in the case
of the 1st layer, the input image) with a set of learned filters; (ii) passing the responses
through a rectified linear function (relu(x) = max(x, 0)); (iii) [optionally] max pooling
over local neighborhoods and (iv) [optionally] a local contrast operation that normalizes
the responses across feature maps - refer to (Krizhevsky et al., 2012) for details. The top
few layers of the network are conventional fully-connected networks and the final layer is a
softmax classifier Zeiler and Fergus (2013).
The model can be trained using a large set of N labeled images {x, y}, where label yi is a
discrete variable indicating the true class. A cross-entropy loss function, suitable for image
classification, is used to compare yˆi and yi . The parameters of the network (filters in the
convolutional layers, weight matrices in the fully-connected layers and biases) are trained
by back-propagating the derivative of the loss with respect to the parameters throughout
the network and updating the parameters via stochastic gradient descent.
Visualizing features in order to gain insight into the behavior of networks has been done
previously, however, limited to the first layer(s). In ”deeper” layers it is not possible to
make projections to the pixel space, consequently there are very few methods to date to
provide interpretability of such networks. For example Erhan et al. (2009) proposed to
find the optimal stimulus for each unit by performing gradient descent in image space to
maximize each unit’s activation. This requires a careful initialization and does not give
any information about the unit’s invariances. Motivated by this shortcoming, Ngiam et al.
(2010) demonstrate how the Hessian of a given unit may be computed numerically around
the optimal response, giving some insight into invariances. The problem still remains that
for higher layers the invariances are extremely complex and can therefore hardly be captured.
A very interesting approach has been developed by Zeiler and Fergus (2013): they
introduce a visualization technique that reveals input stimuli exciting individual feature
maps at any layer in the model, which allows to observe a kind of ”evolution” of features
during the training phase. This allows insight into the internals of the model as well as
to discover problems. Their technique is based on a multi-layered deconvolutional network
(deconvnet) (Zeiler et al., 2011) in order to project the feature activations back to the
input pixel space. The basic principle can be inferred from Figure 5 and Figure 6. The
procedure is ”unpool, rectify, filter” in order to reconstruct the activities in the layer which
was responsible for an activation. This procedure is repeated until the input pixel space is
reached.
12

Explainable Artificial Intelligence in Digital Pathology

Figure 5: A deconvnet layer (left) attached to a convnet layer (right). The deconvnet is
able to reconstruct an approximate version of the convnet features from each layer
beneath; image source: Zeiler and Fergus (2013)

Figure 6: The unpooling operation in the deconvnet, using switches which record the location of the local max in each pooling region (colored zones) during pooling in the
convnet; image source: Zeiler and Fergus (2013)

13

Holzinger, Malle, Kieseberg, Roth, Müller, Reihs, Zatloukal

4. Learning from Heterogeneous Graphs
Graph Theory (Harary et al., 1965; Bondy and Murty, 1976) provides powerful tools to
link data structures of different sources and to reason about probabilistic connections between data objects (Strogatz, 2001). Graphical models (Lauritzen, 1996) combined with
probability theory (Kolmogorov, 1950), which can represent a set of random variables and
conditional dependencies, form the basis for causality (Pearl, 2009); this is of extreme importance towards reaching explainable AI. Recent developments in probabilistic programming
support such approaches (Fadja and Riguzzi, 2017; Tran et al., 2017).
An example is the graph-based approach for mitosis extraction in breast cancer from
WSI, which has been presented by Roullier et al. (2011): Their segmentation uses multiresolution, which reproduces the WSI examination done by a (human) pathologist. Each
resolution level is then analyzed with a focus of attention resulting from a coarser resolution
level analysis. At each level a spatial refinement by label regularization is performed to
obtain a more accurate segmentation. The proposed segmentation is fully unsupervised
by using domain specific knowledge and their strategy for multi-resolution segmentation
makes use of regularization on graphs both for image simplification and segmentation. This
approach is based on an interesting nonlocal discrete regularization framework on weighted
graphs of the arbitrary topologies for image and manifold processing (Elmoataz et al., 2008),
where graphs are the shared representation.
However, the challenge of extracting appropriate graphs out of ”natural images” remains,
especially w.r.t. identifiable biological structures, which is not an easy task (Holzinger et al.,
2014c,b).
With respect to the sheer size of our pathological input space (see Figure 8) we propose
an intriguing possibility of graph-theory: To project an image’s pixel space down to a graph’s
vertex-space, in which each vertex represents a Region-of-Interest (ROI) within the original
image (e.g. a cell nucleus). Given a meaningful representation, we can subsequently apply
powerful graph algorithms including belief-propagation or Bayesian networks - or even train
CNNs using the vertex vector / adjacency matrix as input layer (see Figure 7).

Figure 7: A possible graph computational pipeline: 1) Image preprocessing & segmentation,
2) projection from pixel to vertex space, 3) graph construction, 4) graph-based
data mining, 5) classification

While the real power of graph theory does not lie in constructing graphs from individual
items but in interlinking heterogeneous sources, simply connecting EHR, *omics and image
data of one single patient does not present us with an ”interesting” graph, since almost no
14

Explainable Artificial Intelligence in Digital Pathology

information will be encoded by the graph structure itself (i.e. the topology of the graph),
but remain within the ”nodes” representing the original data. Therefore, it is essential
to devise methodologies which allow us to extract features from different data modalities
that are atomic, isolated, meaningful and - above all - linkable in order to produce a data
structure amenable to graph-theoretical procedures. This necessitates the development of
ashared representation on the feature level, e.g. the mapping of features from different
domains to a shared concept space.
While this could be achieved via Word embeddings for EHRs (Word2Vec) and via random walks on image subgraphs / graphs extracted from subimages (Node2Vec), it is yet
unclear how to properly perform meaningful embeddings for *omics data. In any case, the
goal of such projections is to make data fragments from different sources comparable in
terms of structural as well as functional similarity and importance to specific use cases.
This could serve as a first step towards explainability, as concept-mappings over modalities
would enable text-and-omics-to-screen reasoning algorithms which form part of the idea of
a holistic user interface.
Finally, merging heterogeneous data sources provides the intriguing possibility of forming superior models about patterns of interplay between symptoms / characteristics on the
cellular, *omics, and diagnostic level. Introducing such parameter schemes to generative adversarial networks (GANs) might even allow for the generation of quasi-realistic, synthetic
patient data and thus the provision of virtual pathology data sets.

5. On Topological Data Mining from Medical Images
Much potential lies in topological data mining (Holzinger, 2014b), where we need point cloud
data sets or distances as inputs. A set of such primitives forms a space, and if we have finite
sets equipped with proximity or similarity functions simq : S q+1 → [0, 1], which measure
how “close” or “similar” (q + 1)-tuples of elements of S are, we speak about a topological
space. A value of 0 means totally different objects, while 1 corresponds to equivalent items.
This is highly relevant for explainable AI, because natural data form certain structures
called manifolds (Abraham et al., 1988; Edelsbrunner et al., 2001). Manifolds can be seen
as a topological space that is locally homeomorphic (that means it has a continuous function
with an inverse function) to a real n-dimensional space. In other words: X is a d -manifold if
every point of X has a neighborhood homeomorphic to Bd ; it is called bounded if every point
has a neighborhood homeomorphic to B or Bd+ (Cannon, 1978). This is highly relevant for
many applications in machine learning where we are confronted with high-dimensional data
and to find meaningful representations in the lower-dimensions (Tenenbaum et al., 2000;
Saul and Roweis, 2003; Weinberger and Saul, 2006). A topological space can therefor be
seen as an abstraction of a metric space, and similarly, manifolds generalize the connectivity
of d-dimensional Euclidean spaces Bd by being locally similar but globally different. A ddimensional chart at p ∈ X is a homeomorphism φ : U → Rd onto an open subset of Rd ,
where U is a neighborhood of p and open is defined using the metric. A d-dimensional
manifold (d-manifold) is a topological space X with a d-dimensional chart at every point
x ∈ X (Zomorodian, 2010).
Also interesting for our work are simplicial complexes (”simplicials”) which are spaces
described in a very particular way, the basis of which is homology (Cerri et al., 2013).
15

Holzinger, Malle, Kieseberg, Roth, Müller, Reihs, Zatloukal

The reason for their existence is that it is not possible to represent surfaces precisely in
a computer system due to limited computational storage; thus, surfaces are sampled and
represented with triangulations. Such a triangulation is called a simplicial complex, and
is a combinatorial space that can represent a space. With such simplicial complexes, the
topology of a space can be separated from its geometry.
One way to create a simplicial complex is to examine all subsets of points, and if any
subsets of points are close enough, a p-simplex (e.g. line) is added to the complex with
those points as vertices. For instance, a Vietoris-Rips complex of diameter  is defined
as V R() = σ|diam(σ) ≤ , where diam() is defined as the largest distance between two
points in σ. A common way to analyze topological structures is to apply persistent homology
(Zomorodian and Carlsson, 2005), which identifies cluster, holes and voids. It is assumed
that more robust topological structures are the one which persist with increasing . For
us it is very relevant to extract significant features, thus these methods are useful, since
they provide robust and general feature definitions with emphasis on global information,
e.g. Alpha Shapes (Edelsbrunner and Mücke, 1994).
The combination of topology and machine learning towards geometric machine learning
could be very useful in the future, e.g. geometric deep learning, which attempts to generalize
structured deep learning models to non-Euclidean domains, such as graphs and manifolds
(Lee et al., 2016; Bronstein et al., 2017) and help in developing explainable-AI.

6. Interactive Machine Learning with the Expert-in-the-Loop
The central challenge in machine learning is the development of a model which relates
observations X to target variables Y = x1 , x2 , x3 , ..., inferring either decisions or predictions.
While traditionally we deal with deterministic values for Y , in some domains - such as
pathology - we need to consider uncertainties and therefore seek applications where Y
includes unknowns or rare categories - and where the application of automatic machine
learning approaches may result in the danger of modeling artifacts.
A further problem is that the complexity of automated machine learning algorithms
has generally detained non-experts from the application of such solutions. However, for
solving real-world problems, the integration of a domain expert’s knowledge can sometimes
be indispensable and the interaction of the domain expert with the data would greatly
enhance the knowledge discovery process, consequently iML puts the pathologist-in-theloop to enable what neither a human nor a computer could do on their own (Holzinger,
2016b).
One approach to mitigate these shortcomings is to involve humans in the model-building
effort; traditionally this has been restricted to having people label data in a pre-processing
stage or by requesting experts to select or weight features (bias-injection) before the actual
Machine Learning happens. A real interactive approach though requires the interplay of
algorithmic action and human feedback in a continuous loop, thereby enabling the algorithm
to adapt it’s internal learning strategy on-the-fly, thus effectively opening up the black box
approach (Holzinger, 2016b,a) to a glass box6 .
This method brings with it two major benefits: 1) for problems in high-dimensional
search spaces, it allows the algorithm to restrict possibilities by making use of human
6. Our project page is accessible via: https://hci-kdd.org/project/iml

16

Explainable Artificial Intelligence in Digital Pathology

Figure 8: FFPE section of a human hepatocellular carcinoma, hematoxylin and eosin staining; comparison of scans with 0,24 micrometer/pixel and 0,12 micrometer/pixel
(Image Source: Pathology Graz, obtained with the new P1000 scanner of 3DHistech)

experience translated into heuristic functions, and 2) as decisions are always - at least partly
- based on inputs stemming from the realm of human expertise, any subsequently automated
ML approach will exhibit the same patterns of reasoning, which makes algorithmic decisions
much more explainable from a human perspective.
An interesting aspect of such interaction is the Bayesian manner by which humans
assign significance to data points they observe and the hypotheses they extract from such
data; instead of seeing ”importance of data” and ”hypothesis” as separate entities, one
should more appropriately model the mutual influence as ”importance of data, given some
hypothesis” and ”hypothesis, given some important data points”.
17

Holzinger, Malle, Kieseberg, Roth, Müller, Reihs, Zatloukal

In combination with heterogeneous data graphs and a holistic user interface, we can
conceive of a progressive interaction cycle where reasoning algorithms present the pathologist with their insights in the form of recommendations (e.g. diagnosis, prognosis, valuable
related information) which a pathologist could accept or reject (see Figure 7, providing
continuous feedback for the algorithm to change it’s internal mapping parameters. This
feedback could be collected implicitly via affective computing, i.e. the observation of a user’s
reaction to content without eliciting conscious, deliberate feedback. This can be done by
microscope eye-tracking (Duchowski, 2007) which can provide novel insights into the analysis behavior of a pathologist (Jaarsma et al., 2014; Krupinski et al., 2006); comparison
studies can be enormously useful for the machine learning community7 .
Projecting this approach further into the future, we could go beyond gaining insights
from moment-to-moment user observation to a system of intention mining, where a chain
of user (inter)actions is utilized to derive a pathologist’s individual strategy of dealing with
a given problem when presented specific observations. The resulting models of individual
learning strategies could be analyzed w.r.t. to their historic success, beneficial strategy
phases could be extracted from each expert’s profile and merged into a virtual pathologist
combining the best traits of each human archetype. Although the practical deployment of
such a ”super pathologist” might evoke negative reactions from medical experts and patients
alike, it would present an ideal solution in education and training of future practitioners
(especially given the shortage of expert personnel in the field), and contribute to the research
in AI generally and ML specifically.

7. Explainability of Machine Decisions
Being a rather general term, explainability is usually defined as making automated ML
decisions transparent, which one can decompose into meaning interpretable (what effect
was that decision supposed to have?), comprehensible (which data led to the decision?)
and reproducible (given the data and specific circumstances, can we model a function that
outputs the same decision?). In order to achieve this, we need to identify and disentangle
explanatory factors in the underlying data, may it be pixels in images, tokens in text, or
base-pairs in genetic sequences.
Apart from several references to explainability in earlier sections, a few generic approaches have been recently proposed which in our view warrant specific presentation. In
Ribeiro et al. (2016) the authors introduce a model they call LIME - Local Interpretable
Model Explanation - which aims to explain any classification or prediction result by approximating it locally with an interpretable model. They achieve this via sampling around
local instances until they arrive at a linear approximation of the global decision function
which behaves in a locally faithful way.
A very recent work Bach et al. (2015) describes layer-wise relevance propagation (LRP)
which the authors claim applicable for any form of layered computation architecture. The
method preserves constraints to a total relevance score per layer, so that input dimensions
”compete” for their influence on the next layer. The system works by first computing the
overall function and then working backwards via distributing the relevance of a neuron k at
layer l + 1 onto its input neurons at layer l all the way back to the input dimensions (e.g.
7. Compare to the work done by Gopnik and et al. (2004)

18

Explainable Artificial Intelligence in Digital Pathology

pixels within an image). These can then be visualized via heat maps, offering an intuitive
interface for human comprehension.
An interesting application framework for explainable interactive ML is Bonsai 8 , which
allows programmers without extensive ML experience to train so-called ”curricula”, i.e.
bottom-up learning strategies for particular tasks in a Domain Specific Language. These
programs are separated into different learning sections and phases, so that the user acts as
a constraint-giver to the solution space during training (which has the downside of limiting
the universal learning-capabilities of deep architectures), so that mis-classifications of the
trained model in the testing phase can be clearly attributed to a ”sub-program”, thereby
helping explainability and model acceptance.

8. Privacy Aware Machine Learning
The General Data Protection Regulation (GDPR and ISO/IEC 27001) coming into effect on May 25th 2018 attracts a lot of attention in the medical area, as it is concerned
with all major issues regarding the processing of personal sensitive information and deals
with the legal requirements for data collection, consent regarding processing, anonymization/pseudonymization, data storage, transparency and deletion (”the right to be forgotten”
(Malle et al., 2016), calling for federated learning approaches (Malle et al., 2017)).
Still, the major issue is that many details are currently not defined, e.g. whether deletion
needs to be done on a physical or simply a logical level, or how strong the anonymizationfactors need to be (Villaronga et al., 2017).
Furthermore, some parts are formulated in a way that cannot be achieved with current technological means, e.g. de-anonymization being impossible in any case, as well as
the antagonism between deletion and transparency. Thus, the issue of securing sensitive
information is one of the big challenges in machine learning in health related environments.
8.1 Digital Self-Determination
Providing digital self-determination is one of the major targets of the GDPR and forms
the background behind several aspects like (i) the requirement for consent, (ii) no consent
given implying the need for anonymization, (iii) transparency of the whole analysis process,
including the right of the data subject to promptly receive all information regarding the
processing of his/her data, as well as (iv) the right to rectify any erroneous information
entered into the processing system, even granting (v) the right to have all personal data
deleted. This ”right to be forgotten” can even be called upon in case explicit consent was
granted beforehand, meaning consent can be revoked by the data subject basically at all
times.
The main problem behind deletion, which is also true for rectification of erroneous data,
is that there is currently no definition of the word ”deletion” within the GDPR( Villaronga
et al. (2017)). Furthermore, the GDPR is written as a legal text with a lot of absolutes
that can be interpreted as to demand things that are technically impossible like physical
deletion in cloud environments. Another issue regarding deletion are backups that must
8. https://bons.ai

19

Holzinger, Malle, Kieseberg, Roth, Müller, Reihs, Zatloukal

Figure 9: Example for a B + − T ree
not be changed with respect to backup policies and for security reasons. Also the right to
transparency can pose a severe problem, as traceability in complex enrichment processes
might require complex Audit & Control mechanisms that will contain parts of the user data.
While in theory, deleting data from a system is not considered a major problem, in the
reality of complex systems provable deletion, i.e. being able to guarantee that the deleted
data cannot be restored, is far from trivial (Villaronga et al., 2017). This becomes especially
problematic when considering complex systems like databases that on the one hand store
most of the data in real-life environments and are therefore the main concern for data
deletion and on the other hand have to cater to requirements like ACID-compliance (Haerder
and Reuter, 1983).
While manipulating the data requires quite some knowledge of the internal mechanisms
of the DBMS in question, in the end it is just a legal question on how much resilience a
deletion mechanism has to provide. To further illustrate this aspect, we stay within the
database environment and have a look at ”normal” deletion: Quite like in most file systems,
data is not actually deleted from the disk, e.g. by overwriting the memory, but just unlinked
from the global search indices: Typically, relational databases store their content inside a
special structure called a B + -tree (Bayer and McCreight, 2002) (see Figure 9 for an example,
still, reality is a bit more complex, but this simplification does not significantly change our
point):
• For the number mi of elements of node i holds d2 ≤ i ≤ d, where d is a pre-defined
value (the order) for the whole tree. Only the root node is allowed to possess less
elements (> 0). Inside the nodes, the elements are stored as sorted lists.
• Each inner node holding m elements possesses m + 1 child nodes.
• The inner nodes of the B + -tree do not store actual information on the elements,
but just the information required to let the search engine navigate through the tree
efficiently. All actual data is stored in the leafs, which are all on the same level in the
tree (balanced tree).

8.2 Unsolicited Propagation of Data
Within a lot of research environments, cooperation with external experts or with academic
peers is very important. Within many data driven research approaches this sooner or later
20

Explainable Artificial Intelligence in Digital Pathology

leads to the topic of data exchange, e.g. by sharing data within a common environment
or providing experts with sample data. While regulations and legal frameworks like the
GDPR deal with the topic of protecting the data subjects, another issue is often neglected:
The inherent value of even well-anonymized data, or data sets that do not contain sensitive
personal information at all. While not a Privacy issue, a lot of data that is usable enough
for providing insight on a certain topic does possess an inherent value that needs to be protected, as it is often a vital asset of the (academic/medical) institution. This is especially
true for large collections of pathological data that have been obtained through an expensive
process of physical labor. Thus, when institutions join cooperative research projects, they
must make sure to be able to protect these data assets.
Typical measures proposed by traditional security research mainly fall into the category of proactive measures, typically limiting the actual data exchange. Examples for these
proactive measures include the setup of sealed research environments, typically virtualized
environments containing all required data and the analysis tools while not allowing the
extraction of detailed information, or splitting the information over several independent
processing entities that are assumed to not collude (e.g. Hudic et al. (2013)), or providing
data only in anonymized or aggregated form.
Proactive measures are of course massively interesting but cannot be used under all
circumstances, e.g. when the analysts require their own analysis environments or need the
data with a certain quality. Reactive approaches on the other hand do not aim at limiting
unsolicited propagation, but rather allow to detect it, thus allowing the data owner to
recoup their losses with legal actions. Fingerprinting/watermarking of data is the most
popular approach in this area. Fingerprinted data contains certain changes to the original
information that allow the data owner to determine and prove which partner was the source
of a data leak. These fingerprints are required to be stable against colluding attackers and
it must not be possible to identify and remove the marks from a marked set. Furthermore,
they must be unforgeable in order to protect innocent partners against wrongful accusations.
In addition, the less information is required to detect the data leak, the better. While many
fingerprinting approaches rely on the addition or change of data records, there are also
approaches that utilize the intrinsic features of anonymization (Kieseberg et al., 2014),
which can be very useful in case the data has to be anonymized anyways.
8.3 Manipulation Detection
Especially when considering expert-in-the-loop systems in the medical area, the question
of securing information against manipulation gains increasing importance. In case experts
are used to control and steer the decision making process, their input might be responsible
for the well-being of patients, while used in a way that makes it hard for them to actually
control the effects, as the data is taken as input by machine learning algorithms and utilized
as e.g. knowledge base.
Thus, one vital aspect lies in gaining trust of the doctor in the loop (Kieseberg et al.,
2016b,a) i.e. assuring that the information that was presented as starting point for giving
expertise is not changed afterwards. In addition, the expert must be sure that the expertise
21

Holzinger, Malle, Kieseberg, Roth, Müller, Reihs, Zatloukal

Figure 10: Chaining log entries

(the information) gathered from him has not been distorted, i.e. in case the algorithms
produce wrong results, the blame cannot be shifted towards him/her without justification.
This is especially important in order to protect the experts against cover-ups, e.g. other
experts or the developers of the machine learning algorithms trying to put the blame towards them. In our experience, providing such mechanisms is of vital importance in order
to attract expert involvement and make them share their knowledge, as well as to increase
the acceptance of the results of semi-automated decision engines.
While many of the tasks required to establish a secure environment harken back to
typical topics of system hardening against inside and outside attacks, i.e. making it as impenetrable as possible, and are therefore highly dependent on the actual technical system
in place, the data needs to be secured against another attack vector typically not included
in security assessments: The manipulation of the data by the system itself, i.e. typically
implemented in the form of a database management system. While the system and its administrators are typically trusted to a certain extent in most architectures, they need to be
considered as a major antagonist for data manipulation in order to cover up the application
of wrong algorithms or the provisioning of erroneous data. Thus, every piece of information
that is used in the analysis needs to be secured against subsequent manipulation attacks,
even from users possessing administrator privileges.
In addition, other approaches have been introduced including utilization of the database
replication mechanism etc., still, research in this area is rather at its infancy and has not received much attention in the past as most real-life systems use traditional system hardening
measures together with organizational measures.
22

Explainable Artificial Intelligence in Digital Pathology

9. Conclusion and Future Outlook
Digital pathology poses manifold challenges for AI research generally and ML research
specifically. Solutions will benefit education, training, research and clinical decision support.
The setting of digital pathology is ideally suited to study both human learning and human
decision making and contrast it to machine learning and machine decision making. By
comparing both the strengths/weaknesses of machine as well as human intelligence it will
be possible to find solutions where we are currently lacking appropriate methods.
Future medicine relies increasingly on various data for better detection and treatment of
diseases. In order to extract knowledge from massive amounts of information contained in
high resolution images, AI and ML methods are indispensable. Moreover, digital pathology
provides a fertile environment to integrate data from heterogeneous and distributed sources
(images, text, omics), making novel information accessible and quantifiable - which is not
accessible and quantifiable for the human expert to date. A major challenge in this area
will be to ”break down” the constituent parts of each data source (image region, gene subsequence, EHR entry) and to make them interlinkable on a level allowing for reasoning
between sensory records, image anomalies & textual description of symptoms. Eventually,
this will enable completely new knowledge and query-based systems, where experts may
input an image region and receive genetic sequences potentially explaining the observation.
AI-augmented digital pathology will change the education and training of pathologists,
which is an urgently needed solution to the global shortage of medical specialists. In order
to realize this potential, we will have to define interaction points between pathologists and
the learning system; this entails studies to discover which internal learning states can be
best represented in a human-intelligible way, what kind of problems the expert is asked to
decide (accept/reject a final recommendation or help identifying image regions), and how
to incorporate the human decision back into the internal machine model. Defining a new
standard protocol for such interactions would also enable external software providers to
hook into the process, generating new business models for interactive diagnostics.
Providing insights into the inner workings of algorithms is not only a future legal requirement but also a valuable tool for ”debugging”, especially in the case of deep neural
architectures where exact coefficients at exact moments are to date nearly uninterpretable
by human experts. Building on recent works we will in future work experiment with the
interplay of iML and explainability - injecting search space constraints based on user decisions gathered from an interaction loop into our programs. Consequently, it is necessary
to contrast any gains in explainability with the potential loss of generality incurred by the
restriction of algorithmic freedom.
Although deep learning has long proven it’s worth in a multitude of ML challenges, we
still need to make it efficient on extremely large images (such as WSI). Our main challenge
will lie in finding more compact, yet smart image representations e.g. via graph extraction,
offering the possibility to ”asymmetrically” compress an image via region-to-node projection, i.e. applying different compression rates to regions of different importance. As neural
nets take simple feature vectors as their inputs, we can easily replace a pixel representation with a node feature vector or the entries of an adjacency / Laplacian matrix. This
approach also promises insights above and beyond image classification, since in the process
we will experiment with DNNs ”intuiting” network properties like degree distributions or
23

Holzinger, Malle, Kieseberg, Roth, Müller, Reihs, Zatloukal

centralities, potentially yielding deep approximate graph algorithms for problems whose
algorithmic runtime has not improved in decades.
As modern AI/ML approaches require great amounts of data, training on local databases
could easily hit its limits, yet legal requirements as well as societal expectations restrict us
from freely sharing vulnerable data. In order to overcome the unsatisfying trade-off we
are confronted with today - lose precision via learning on anonymized data, or risk severe
repercussions, we need to examine methods to learn from other models instead of other
data.
Eventually, we strive to combine parallel & distributed DNNs working within their
respective ”local sphere” into a powerful network of learners, each fueled by their unique,
private knowledge bases yet capable of incorporating model insights from their peer learners
- achieving the grand goal of secure, efficient and reliable privacy aware federated learning.
Acknowledgments
We are grateful for valuable discussions and comments from our local, national and international colleagues. We thank our industrial partner 3DHistech for their assistance with
our new scanner and the Biobank Graz for valuable support. Acknowledged Funding:
Hochschulraum-Strukturmittelverordnung.

References
Ralph Abraham, Jerrold E. Marsden, and Tudor Ratiu. Manifolds, tensor analysis, and
applications. Springer, New York, 1988. doi: 10.1007/978-1-4612-1029-0.
Asmara Afework, Michael D. Beynon, Fabian Bustamante, Soon Cho, Angelo Demarzo,
Renato Ferreira, Robert Miller, Mark Silberman, Joel Saltz, and Alan Sussman. Digital
dynamic telepathology - the virtual microscope. In Proceedings of the AMIA Symposium,
pages 912–916. American Medical Informatics Association, 1998.
Shaimaa Al-Janabi, André Huisman, and Paul J. Van Diest. Digital pathology: current
status and future perspectives. Histopathology, 61(1):1–9, 2012. doi: 10.1111/j.1365-2559.
2011.03814.x.
Itamar Arel, Derek C. Rose, and Thomas P. Karnowski. Deep machine learning - a new frontier in artificial intelligence research [research frontier]. IEEE Computational Intelligence
Magazine, 5(4):13–18, 2010. doi: 10.1109/MCI.2010.938364.
Carlos Arteta, Victor Lempitsky, J. Alison Noble, and Andrew Zisserman. Learning to
detect cells using non-overlapping extremal regions. In International Conference on
Medical Image Computing and Computer-Assisted Intervention MICCAI, pages 348–356.
Springer, Heidelberg, 2012. doi: 10.1007/978-3-642-33415-3 43.
Naoki Asada and Kunio Doi. Method and system for differential diagnosis based on clinical
and radiological information using artificial neural networks. Patent US5622171 A, 1997.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert
Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions
24

Explainable Artificial Intelligence in Digital Pathology

by layer-wise relevance propagation. PloS one, 10(7):e0130140, 2015. doi: 10.1371/
journal.pone.0130140.
Mattia Barbareschi, Francesca Demichelis, Stefano Forti, and Paolo Dalla Palma. Digital
pathology: Science fiction? International journal of surgical pathology, 8(4):261–263,
2000.
Laura Barisoni, Charlotte Gimpel, Renate Kain, Arvydas Laurinavicius, Gloria Bueno,
Caihong Zeng, Zhihong Liu, Franz Schaefer, Matthias Kretzler, and Lawrence B. Holzman. Digital pathology imaging as a novel platform for standardization and globalization
of quantitative nephropathology. Clinical Kidney Journal, 10(2):176–187, 2017. doi:
10.1093/ckj/sfw129.
Rudolf Bayer and Edward McCreight. Organization and maintenance of large ordered indexes. In Software pioneers, pages 245–262. Springer, 2002. doi: 10.1007/
978-3-642-59412-0 15.
Lionel Blanchet and Agnieszka Smolinska. Data fusion in metabolomics and proteomics
for biomarker discovery. In Klaus Jung, editor, Statistical Analysis in Proteomics, pages
209–223. Springer, New York, 2016. doi: 10.1007/978-1-4939-3106-4 14.
Marcus D Bloice, Christof Stocker, and Andreas Holzinger. Augmentor: An image augmentation library for machine learning. arXiv preprint arXiv:1708.04680, 2017.
John A. Bondy and Uppaluri S. R. Murty. Graph theory with applications. Elsevier, New
York, Amsterdam, Oxford, 1976.
Michael M. Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst.
Geometric deep learning: Going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18–42, 2017. doi: 10.1109/MSP.2017.2693418.
Jinzheng Cai, Le Lu, Yuanpu Xie, Fuyong Xing, and Lin Yang. Pancreas segmentation
in mri using graph-based decision fusion on convolutional neural networks. In International Conference on Medical Image Computing and Computer-Assisted Intervention
(MICCAI), Lecture Notes in Computer Science LNCS 10435, pages 674–682. Springer,
Cham, 2017. doi: 10.1007/978-3-319-66179-7 77.
James W. Cannon. The recognition problem: what is a topological manifold? Bulletin of
the American Mathematical Society, 84(5):832–866, 1978.
Andrea Cerri, Barbara Di Fabio, Massimo Ferri, Patrizio Frosini, and Claudia Landi. Betti
numbers in multidimensional persistent homology are stable functions. Mathematical
Methods in the Applied Sciences, 36(12):1543–1557, 2013. doi: 10.1002/mma.2704.
Fred R. Dee. Virtual microscopy in pathology education. Human pathology, 40(8):1112–
1121, 2009.
Andrew T. Duchowski. Eye tracking methodology: Theory and practice. Springer, Cham,
2007. doi: 10.1007/978-3-319-57883-5.
25

Holzinger, Malle, Kieseberg, Roth, Müller, Reihs, Zatloukal

Herbert Edelsbrunner and Ernst P. Mücke. 3-dimensional alpha-shapes. ACM Transactions
on Graphics, 13(1):43–72, 1994. doi: 10.1145/174462.156635.
Herbert Edelsbrunner, John Harer, and Afra Zomorodian. Hierarchical morse complexes
for piecewise linear 2-manifolds. In Proceedings of the seventeenth annual symposium on
Computational geometry, pages 70–79. ACM, 2001. doi: 10.1145/378583.378626.
Abderrahim Elmoataz, Olivier Lézoray, and Sébastien Bougleux. Nonlocal discrete regularization on weighted graphs: A framework for image and manifold processing. IEEE
Transactions on Image Processing, 17(7):1047–1060, 2008. doi: 10.1109/TIP.2008.924284.
Douglas C. Engelbart. Toward augmenting the human intellect and boosting our collective
IQ. Communications of the ACM, 38(8):30–32, 1995. doi: 10.1145/208344.208352.
Douglas C. Engelbart and William K. English. A research center for augmenting human
intellect. In AFIPS 68, Fall joint computer conference, pages 395–410. ACM, 1968. doi:
10.1145/1476589.1476645.
Jonathan I. Epstein, Patrick C. Walsh, and Fred Sanfilippo. Clinical and cost impact of
second-opinion pathology: Review of prostate biopsies prior to radical prostatectomy.
The American Journal of Surgical Pathology, 20(7):851–857, 1996.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higherlayer features of a deep network. University of Montreal Technical Report Nr. 1341,
2009.
Andre Esteva, Brett Kuprel, Roberto A. Novoa, Justin Ko, Susan M. Swetter, Helen M.
Blau, and Sebastian Thrun. Dermatologist-level classification of skin cancer with deep
neural networks. Nature, 542(7639):115–118, 2017. doi: 10.1038/nature21056.
Arnaud N. Fadja and Fabrizio Riguzzi. Probabilistic logic programming in action. In
Andreas Holzinger, Randy Goebel, Massimo Ferri, and Vasile Palade, editors, Towards
Integrative Machine Learning and Knowledge Extraction: BIRS Workshop, Banff, AB,
Canada, July 24-26, 2015, Revised Selected Papers, pages 89–116. Springer, Cham, 2017.
doi: 10.1007/978-3-319-69775-8 5.
Günther Gell, Miroslav Madjaric, Werner Leodolter, Wolfgang Köle, and Hubert Leitner.
HIS purchase projects in public hospitals of styria, austria. International Journal of
Medical Informatics, 58-59:147–155, 2000. doi: 10.1016/S1386-5056(00)00083-6.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil
Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Zhoubin
Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger,
editors, Advances in neural information processing systems (NIPS), pages 2672–2680,
2014.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, Cambridge (MA), 2016.
26

Explainable Artificial Intelligence in Digital Pathology

Alison Gopnik and et al. A theory of causal learning in children: causal maps and Bayes
nets. Psychological review, 111(1):3–32, 2004. doi: 10.1037/0033-295X.111.1.3.
Hayit Greenspan, Bram van Ginneken, and Ronald M. Summers. Guest editorial deep
learning in medical imaging: Overview and future promise of an exciting new technique.
IEEE Transactions on Medical Imaging, 35(5):1153–1159, 2016. doi: 10.1109/TMI.2016.
2553401.
Theo Haerder and Andreas Reuter. Principles of transaction-oriented database recovery.
ACM Computing Surveys (CSUR), 15(4):287–317, 1983. doi: 10.1145/289.291.
Pierre Hainaut, Jim Vaught, Kurt Zatloukal, and Markus Pasterk. Biobanking of Human
Biospecimens. Springer, Cham, 2017. doi: 10.1007/978-3-319-55120-3.
Frank Harary, Robert Z. Norman, and Dorwin Cartwright. Structural Models: An Introduction to the Theory of Directed Graphs. Wiley, New York, 1965.
Søren Hauberg, Oren Freifeld, Anders Boesen Lindbo Larsen, John Fisher, and Lars Hansen.
Dreaming more data: Class-dependent distributions over diffeomorphisms for learned
data augmentation. In 19th International Conference on Artificial Intelligence and Statistics (AISTATS), pages 342–350. JMLR, 2016.
Andreas Holzinger. On Knowledge Discovery and Interactive Intelligent Visualization of
Biomedical Data - Challenges in Human-Computer Interaction & Biomedical Informatics. In Markus Helfert, Chiara Fancalanci, and Joaquim Filipe, editors, DATA 2012,
International Conference on Data Technologies and Applications, pages 5–16. 2012.
Andreas Holzinger. Human-Computer Interaction and Knowledge Discovery (HCI-KDD):
What is the benefit of bringing those two fields to work together? In Alfredo Cuzzocrea,
Christian Kittl, Dimitris E. Simos, Edgar Weippl, and Lida Xu, editors, Multidisciplinary
Research and Practice for Information Systems, Springer Lecture Notes in Computer
Science LNCS 8127, pages 319–328. Springer, Heidelberg, Berlin, New York, 2013. doi:
10.1007/978-3-642-40511-2 22.
Andreas Holzinger. Biomedical Informatics: Discovering Knowledge in Big Data. Springer,
New York, 2014a. doi: 10.1007/978-3-319-04528-3.
Andreas Holzinger. On topological data mining. In Interactive Knowledge Discovery and
Data Mining in Biomedical Informatics, Lecture Notes in Computer Science LNCS 8401,
pages 331–356. Springer, Heidelberg, 2014b. doi: 10.1007/978-3-662-43968-5 19.
Andreas Holzinger. Trends in interactive knowledge discovery for personalized medicine:
Cognitive science meets machine learning. IEEE Intelligent Informatics Bulletin, 15(1):
6–14, 2014c.
Andreas Holzinger. Interactive machine learning (iML). Informatik Spektrum, 39(1):64–68,
2016a. doi: 10.1007/s00287-015-0941-6.
27

Holzinger, Malle, Kieseberg, Roth, Müller, Reihs, Zatloukal

Andreas Holzinger. Interactive machine learning for health informatics: When do we
need the human-in-the-loop? Brain Informatics, 3(2):119–131, 2016b. doi: 10.1007/
s40708-016-0042-6.
Andreas Holzinger. Introduction to machine learning and knowledge extraction (MAKE).
Machine Learning and Knowledge Extraction, 1(1):1–20, 2017.
doi: 10.3390/
make1010001.
Andreas Holzinger, Andreas Kainz, Günther Gell, Max Brunold, and Hermann Maurer.
Interactive computer assisted formulation of retrieval requests for a medical information system using an intelligent tutoring system. In Jacqueline Bourdeau and Rachelle
Heller, editors, World Conference on Educational Media and Technology, pages 466–
471. Association for the Advancement of Computing in Education (AACE), 2000. URL
https://www.learntechlib.org/p/16109.
Andreas Holzinger, Matthias Dehmer, and Igor Jurisica. Knowledge discovery and interactive data mining in bioinformatics - state-of-the-art, future challenges and research
directions. BMC Bioinformatics, 15(S6):I1, 2014a. doi: doi:10.1186/1471-2105-15-S6-I1.
Andreas Holzinger, Bernd Malle, Marcus Bloice, Marco Wiltgen, Massimo Ferri, Ignazio
Stanganelli, and Rainer Hofmann-Wellenhof. On the generation of point cloud data
sets: Step one in the knowledge discovery process. In Interactive Knowledge Discovery and Data Mining in Biomedical Informatics, Lecture Notes in Computer Science, LNCS 8401, volume 8401, pages 57–80. Springer, Berlin Heidelberg, 2014b. doi:
10.1007/978-3-662-43968-5 4.
Andreas Holzinger, Bernd Malle, and Nicola Giuliani. On graph extraction from image data.
In Dominik Slezak, James F. Peters, Ah-Hwee Tan, and Lars Schwabe, editors, Brain
Informatics and Health, BIH 2014, Lecture Notes in Artificial Intelligence, LNAI 8609,
pages 552–563. Springer, Heidelberg, Berlin, 2014c. doi: 10.1007/978-3-319-09891-3 50.
Andreas Holzinger, Christof Stocker, and Matthias Dehmer. Big complex biomedical data:
Towards a taxonomy of data. In Mohammad S. Obaidat and Joaquim Filipe, editors,
Communications in Computer and Information Science CCIS 455, pages 3–18. Springer,
Berlin Heidelberg, 2014d. doi: 10.1007/978-3-662-44791-8 1.
Andreas Holzinger, Bernd Malle, Peter Kieseberg, Peter M. Roth, Heimo Müller, Robert
Reihs, and Kurt Zatloukal. Machine learning and knowledge extraction in digital pathology needs an integrative approach. In Towards Integrative Machine Learning and Knowledge Extraction, Springer Lecture Notes in Artificial Intelligence Volume LNAI 10344,
pages 13–50. Springer, Cham, 2017a. doi: 10.1007/978-3-319-69775-8 2.
Andreas Holzinger, Markus Plass, Katharina Holzinger, Gloria Cerasela Crisan, Camelia-M.
Pintea, and Vasile Palade. A glass-box interactive machine learning approach for solving
np-hard problems with the human-in-the-loop. arXiv:1708.01104, 2017b.
Aleksandar Hudic, Shareeful Islam, Peter Kieseberg, Sylvi Rennert, and Edgar Weippl.
Data confidentiality using fragmentation in cloud computing. International Jour28

Explainable Artificial Intelligence in Digital Pathology

nal of Pervasive Computing and Communications, 9(1):37–51, 2013.
17427371311315743.

doi: 10.1108/

André Huisman, Arnoud Looijen, Steven M. van den Brink, and Paul J. van Diest. Creation
of a fully digital pathology slide archive by high-volume tissue slide scanning. Human
pathology, 41(5):751–757, 2010. doi: 10.1016/j.humpath.2009.08.026.
Berthold Huppertz and Andreas Holzinger. Biobanks a source of large biological data sets:
Open problems and future challenges. In Andreas Holzinger and Igor Jurisica, editors,
Interactive Knowledge Discovery and Data Mining in Biomedical Informatics, Lecture
Notes in Computer Science LNCS 8401, pages 317–330. Springer, Berlin, Heidelberg,
2014. doi: 10.1007/978-3-662-43968-5 18.
Thomas Jaarsma, Halszka Jarodzka, Marius Nap, Jeroen J. Merrienboer, and Henny
Boshuizen. Expertise under the microscope: processing histopathological slides. Medical education, 48(3):292–300, 2014. doi: 10.1111/medu.12385.
Fleur Jeanquartier, Claire Jean-Quartier, Tobias Schreck, David Cemernek, and Andreas
Holzinger. Integrating open data on cancer in support to tumor growth analysis. In
Elena M. Renda, Miroslav Bursa, Andreas Holzinger, and Sami Khuri, editors, Lecture
Notes in Computer Science LNCS 9832, pages 49–66. Springer, Cham, 2016. doi: 10.
1007/978-3-319-43949-5 4.
Douglas B. Kell. Metabolomics and systems biology: making sense of the soup. Current
opinion in microbiology, 7(3):296–307, 2004. doi: 10.1016/j.mib.2004.04.012.
Peter Kieseberg, Heidelinde Hobel, Sebastian Schrittwieser, Edgar Weippl, and Andreas
Holzinger. Protecting anonymity in data-driven biomedical science. In Andreas Holzinger
and Igor Jurisica, editors, Interactive Knowledge Discovery and Data Mining in Biomedical Informatics, Lecture Notes in Computer Science, LNCS 8401, pages 301–316.
Springer, Berlin Heidelberg, 2014. doi: 10.1007/978-3-662-43968-5 17.
Peter Kieseberg, Bernd Malle, Peter Frühwirt, Edgar Weippl, and Andreas Holzinger. A
tamper-proof audit and control system for the doctor in the loop. Brain Informatics, 3
(4):269279, 2016a. doi: 10.1007/s40708-016-0046-2.
Peter Kieseberg, Edgar Weippl, and Andreas Holzinger. Trust for the doctor-in-the-loop.
European Research Consortium for Informatics and Mathematics (ERCIM) News: Tackling Big Data in the Life Sciences, 104(1):32–33, 2016b.
Katja Kircher, Annika Larsson, and Jonas Andersson Hultgren. Tactical driving behavior
with different levels of automation. IEEE Transactions on Intelligent Transportation
Systems, 15(1):158–167, 2014. doi: 10.1109/TITS.2013.2277725.
Toshiji Kobayashi, Osamu Takatani, Nobu Hattori, and Kiyoji Kimura. Differential diagnosis of breast tumors. the sensitivity graded method of ultrasonotomography and clinical
evaluation of its diagnostic accuracy. Cancer, 33(4):940–951, 1974.
Andreǐ N. Kolmogorov. Foundations of the Theory of Probability. Chelsea, Oxford, 1950.
29

Holzinger, Malle, Kieseberg, Roth, Müller, Reihs, Zatloukal

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with
deep convolutional neural networks. In Fernando Pereira, Christopher J.C. Burges, Leon
Bottou, and Kilian Q. Weinberger, editors, Advances in neural information processing
systems (NIPS 2012), pages 1097–1105. NIPS, 2012.
Elizabeth A. Krupinski, Allison A. Tillack, Lynne Richter, Jeffrey T. Henderson, Achyut K.
Bhattacharyya, Katherine M. Scott, Anna R. Graham, Michael R. Descour, John R.
Davis, and Ronald S. Weinstein. Eye-movement study and human performance using
telepathology virtual slides. implications for medical education and differences with experience. Human pathology, 37(12):1543–1556, 2006. doi: 10.1016/j.humpath.2006.08.024.
Stephane Lafon, Yosi Keller, and Ronald R. Coifman. Data fusion and multicue data matching by diffusion maps. IEEE Transactions on pattern analysis and machine intelligence,
28(11):1784–1797, 2006. doi: 10.1109/TPAMI.2006.223.
Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, and Samuel J. Gershman.
Building machines that learn and think like people. arXiv:1604.00289, 2016.
Steffen L. Lauritzen. Graphical models. Clarendon Press, Oxford, 1996.
Quoc V. Le, Marc’Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S.
Corrado, Jeff Dean, and Andrew Y. Ng. Building high-level features using large scale
unsupervised learning. arXiv:1112.6209, 2011.
Yann LeCun, Bernhard Boser, John S. Denker, Donnie Henderson, Richard E. Howard,
Wayne Hubbard, and Lawrence D. Jackel. Backpropagation applied to handwritten zip
code recognition. Neural computation, 1(4):541–551, 1989. doi: 10.1162/neco.1989.1.4.
541.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):
436–444, 2015. doi: 10.1038/nature14539.
Jaekoo Lee, Hyunjae Kim, Jongsun Lee, and Sungroh Yoon. Intrinsic geometric information
transfer learning on multiple graph-structured datasets. arXiv:1611.04687, 2016.
Maxwell W. Libbrecht and William Stafford Noble. Machine learning applications in genetics and genomics. Nature Reviews Genetics, 16(6):321–332, 2015. doi: 10.1038/nrg3920.
Yun Liu, Krishna Gadepalli, Mohammad Norouzi, George E. Dahl, Timo Kohlberger, Aleksey Boyko, Subhashini Venugopalan, Aleksei Timofeev, Philip Q. Nelson, Greg S. Corrado, Jason D. Hipp, Lily Peng, and Martin C. Stumpe. Detecting cancer metastases on
gigapixel pathology images. arXiv:1703.02442, 2017.
Mingsheng Long, Jianmin Wang, and Michael I. Jordan. Deep transfer learning with joint
adaptation networks. arXiv:1605.06636, 2016.
Tyron Louw, Natasha Merat, and Hamish Jamson. Engaging with highly automated driving:
to be or not to be in the loop. In Eighth International Driving Symposium on Human
Factors in Driver Assessment, Training and Vehicle Design, pages 190–196, 2015.
30

Explainable Artificial Intelligence in Digital Pathology

Anant Madabhushi and George Lee. Image analysis and machine learning in digital pathology: Challenges and opportunities. Medical Image Analysis, 33:170–175, 2016. doi:
10.1016/j.media.2016.06.037.
Bernd Malle, Peter Kieseberg, Edgar Weippl, and Andreas Holzinger. The right to be
forgotten: Towards machine learning on perturbed knowledge bases. In Springer Lecture
Notes in Computer Science LNCS 9817, pages 251–256. Springer, Heidelberg, Berlin,
New York, 2016. doi: 10.1007/978-3-319-45507-5 17.
Bernd Malle, Nicola Giuliani, Peter Kieseberg, and Andreas Holzinger. The more the
merrier - federated learning from local sphere recommendations. In Machine Learning
and Knowledge Extraction, IFIP CD-MAKE, Lecture Notes in Computer Science LNCS
10410, pages 367–374. Springer, Cham, 2017. doi: 10.1007/978-3-319-66808-6 24.
John McCarthy. From here to human-level ai. Artificial Intelligence, 171(18):1174–1182,
2007. doi: 10.1016/j.artint.2007.10.009.
Jason E. McDermott, Jing Wang, Hugh Mitchell, Bobbie-Jo Webb-Robertson, Ryan Hafen,
John Ramey, and Karin D. Rodland. Challenges in biomarker discovery: combining
expert insights with statistical analysis of complex omics data. Expert opinion on medical
diagnostics, 7(1):37–51, 2013. doi: 10.1517/17530059.2012.718329.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou,
Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning.
arXiv:1312.5602, 2013.
Heimo Müller, Robert Reihs, Kurt Zatloukal, Fleur Jeanquartier, Roxana Merino-Martinez,
David van Enckevort, Morris A. Swertz, and Andreas Holzinger. State-of-the-art and
future challenges in the integration of biobank catalogues. In Andreas Holzinger, Carsten
Röcker, and Martina Ziefle, editors, Smart Health, Lecture Notes in Computer Science
LNCS 8700, pages 261–273. Springer, Heidelberg, 2015. doi: 10.1007/978-3-319-16226-3
11.
Jiquan Ngiam, Zhenghao Chen, Daniel Chia, Pang W. Koh, Quoc V. Le, and Andrew Y.
Ng. Tiled convolutional neural networks. In Advances in neural information processing
systems (NIPS), pages 1279–1287. NIPS, 2010.
Dong Nie, Roger Trullo, Jun Lian, Caroline Petitjean, Su Ruan, Qian Wang, and Dinggang Shen. Medical image synthesis with context-aware generative adversarial networks.
In International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), Lecture Notes in Computer Science LNCS 10435, pages 417–425.
Springer, Cham, 2017. doi: 10.1007/978-3-319-66179-7 48.
Shane O’Sullivan, Andreas Holzinger, Kurt Zatloukal, Paulo Saldiva, Muhammad I. Sajid,
and Dominic Wichmann. Machine learning enhanced virtual autopsy. Autopsy Case
Report, 7(4):3–7, 2017. doi: 10.4322/acr.2017.037.
31

Holzinger, Malle, Kieseberg, Roth, Müller, Reihs, Zatloukal

Christian Payer, Darko Štern, Horst Bischof, and Martin Urschler. Regressing heatmaps
for multiple landmark localization using cnns. In International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), Springer Lecture Notes in Computer Science, LNCS 9901, pages 230–238. Springer, 2016. doi:
10.1007/978-3-319-46723-8 27.
Judea Pearl. Causality: Models, Reasoning, and Inference (Second Edition). Cambridge
University Press, Cambridge, 2009.
Maurice Merrick Reeder and Benjamin Felson. Gamuts in radiology: comprehensive lists of
roentgen differential diagnosis. Fourth Edition. Springer, New York, 2003.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should i trust you?: Explaining the predictions of any classifier. In 22nd ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining, pages 1135–1144. ACM, 2016.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks
for biomedical image segmentation. In International Conference on Medical Image
Computing and Computer-Assisted Intervention (MICCAI), Lecture Notes in Computer Science, LNCS 9351, pages 234–241. Springer, Heidelberg, 2015. doi: 10.1007/
978-3-319-24574-4 28.
Vincent Roullier, Olivier Lézoray, Vinh-Thong Ta, and Abderrahim Elmoataz. Multiresolution graph-based analysis of histopathological whole slide images: Application to
mitotic cell extraction and visualization. Computerized Medical Imaging and Graphics,
35(7):603–615, 2011. doi: 10.1016/j.compmedimag.2011.02.005.
Artem Rozantsev, Vincent Lepetit, and Pascal Fua. On rendering synthetic images for
training an object detector. Computer Vision and Image Understanding, 137:24–37,
2015. doi: 10.1016/j.cviu.2014.12.006.
Stuart J. Russell and Peter Norvig. Artificial Intelligence: A modern approach. Prentice
Hall, Englewood Cliffs, 1995.
Lawrence K. Saul and Sam T. Roweis. Think globally, fit locally: unsupervised learning
of low dimensional manifolds. Journal of machine learning research (JMLR), 4:119–155,
2003.
Jürgen Schmidhuber. Deep learning in neural networks: An overview. Neural Networks, 61
(1):85–117, 2015. doi: 10.1016/j.neunet.2014.09.003.
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den
Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever,
Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis
Hassabis. Mastering the game of go with deep neural networks and tree search. Nature,
529(7587):484–489, 2016. doi: 10.1038/nature16961.
32

Explainable Artificial Intelligence in Digital Pathology

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang,
Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen,
Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel,
and Demis Hassabis. Mastering the game of go without human knowledge. Nature, 550
(7676):354–359, 2017. doi: doi:10.1038/nature24270.
Sameer Singh, Marco Tulio Ribeiro, and Carlos Guestrin. Programs as black-box explanations. arXiv:1611.07579, 2016.
Steven H. Strogatz. Exploring complex networks. Nature, 410(6825):268–276, 2001. doi:
10.1038/35065725.
Anna Louise Swan, Ali Mobasheri, David Allaway, Susan Liddell, and Jaume Bacardit.
Application of machine learning to proteomics data: Classification and biomarker identification in postgenomics biology. Omics-a Journal of Integrative Biology, 17(12):595–610,
2013. doi: 10.1089/omi.2013.0017.
Joshua B. Tenenbaum, Vin de Silva, and John C. Langford. A global geometric framework
for nonlinear dimensionality reduction. Science, 290(5500):2319–2323, 2000. doi: 10.
1126/science.290.5500.2319.
Dustin Tran, Matthew D. Hoffman, Rif A. Saurous, Eugene Brevdo, Kevin Murphy, and
David M. Blei. Deep probabilistic programming. arXiv:1701.03757, 2017.
David A. Van Dyk and Xiao-Li Meng. The art of data augmentation. Journal of Computational and Graphical Statistics, 10(1):1–50, 2001. doi: 10.1198/10618600152418584.
Gert-Jan B. van Ommen, Outi Törnwall, Christian Bréchot, Georges Dagher, Joakim Galli,
Kristian Hveem, Ulf Landegren, Claudio Luchinat, Andres Metspalu, Cecilia Nilsson,
Ove V Solesvik, Markus Perola, Jan-Eric Litton, and Kurt Zatloukal. BBMRI-ERIC as
a resource for pharmaceutical and life science industries: the development of biobankbased expert centres. European Journal of Human Genetics, 23(7):893–900, 2015. doi:
10.1038/ejhg.2014.235.
Eduard Fosch Villaronga, Peter Kieseberg, and Tiffany Li. Humans forget, machines remember: Artificial intelligence and the right to be forgotten. Computer Law & Security
Review, 2017. doi: 10.1016/j.clsr.2017.08.007.
Rudolf Virchow. Vorlesungen über Pathologie I: Die Cellular-Pathologie in ihrer Begründung
auf physiologische und pathologische Gewebelehre. 4. Auflage. August Hirschwald, Berlin,
1871.
Théophane Weber, Sébastien Racanière, David P. Reichert, Lars Buesing, Arthur
Guez, Danilo Jimenez Rezende, Adria Puigdomnech Badia, Oriol Vinyals, Nicolas
Heess, and Yujia Li. Imagination-augmented agents for deep reinforcement learning.
arXiv:1707.06203, 2017.
Kilian Q. Weinberger and Lawrence K. Saul. Unsupervised learning of image manifolds
by semidefinite programming. International journal of computer vision (IJCV), 70(1):
77–90, 2006. doi: 10.1007/s11263-005-4939-z.
33

Holzinger, Malle, Kieseberg, Roth, Müller, Reihs, Zatloukal

Ronald S. Weinstein, Ken J. Bloom, and Laura S. Rozek. Telepathology and the networking
of pathology diagnostic services. Archives of pathology & laboratory medicine, 111(7):646–
652, 1987.
Ronald S. Weinstein, Anna R. Graham, Lynne C. Richter, Gail P. Barker, Elizabeth A.
Krupinski, Ana Maria Lopez, Kristine A. Erps, Achyut K. Bhattacharyya, Yukako Yagi,
and John R. Gilbertson. Overview of telepathology, virtual microscopy, and whole slide
imaging: prospects for the future. Human pathology, 40(8):1057–1069, 2009. doi: 10.
1016/j.humpath.2009.04.006.
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks.
arXiv:1311.2901, 2013.
Matthew D. Zeiler, Graham W. Taylor, and Rob Fergus. Adaptive deconvolutional networks
for mid and high level feature learning. In IEEE International Conference on Computer
Vision (ICCV), pages 2018–2025. IEEE, 2011. doi: 10.1109/ICCV.2011.6126474.
Xinhua Zhang, Le Song, Arthur Gretton, and Alex J. Smola.
Kernel measures
of independence for non-iid data.
In Advances in neural information processing systems (NIPS), pages 1937–1944, 2009. URL http://papers.nips.cc/paper/
3440-kernel-measures-of-independence-for-non-iid-data.
Afra Zomorodian. Computational topology. In M. Atallah and M. Blanton, editors, Algorithms and Theory of Computation Handbook, Second Edition, Volume 2: Special Topics
and Techniques, pages 1–31. Chapman and Hall/CRC, Boca Raton (FL), 2010. doi:
doi:10.1201/9781584888215-c3.
Afra Zomorodian and Gunnar Carlsson. Computing persistent homology. Discrete & Computational Geometry, 33(2):249–274, 2005. doi: 10.1007/s00454-004-1146-y.

34

