explainable AI
113404661
Designing Theory Driven User Centric Explainable AI
From healthcare to criminal justice, artificial intelligence (AI) is increasingly supporting high consequence human decisions. This has spurred the field of explainable AI (XAI) This paper seeks to strengthen empirical application specific investigations of XAI by exploring theoretical underpinnings of human decision making, drawing from the fields of philosophy and psychology. In this paper, we propose a conceptual framework for building human centered, decision theory driven XAI based on an extensive review across these fields. Drawing on this framework, we identify pathways along which human cognitive patterns drives needs for building XAI and how XAI can mitigate common cognitive biases. We then put this framework into practice by designing and implementing an explainable clinical diagnostic tool for intensive care phenotyping and conducting a co design exercise with clinicians. Thereafter, we draw insights into how this framework bridges algorithm generated explanations and human decision making theories. Finally, we discuss implications for XAI design and development.
CHI
Danding Wang Qian Yang Ashraf Abdul Brian Y Lim
202160253
Explainable AI: Interpreting, Explaining and Visualizing Deep Learning
In recent years, machine learning (ML) has become a key enabling technology for the sciences and industry. Especially through improvements in methodology, the availability of large databases and increased computational power, today's ML algorithms are able to achieve excellent performance (at times even exceeding the human level) on an increasing number of complex tasks. Deep learning models are at the forefront of this development. However, due to their nested nonlinear structure, these powerful models have been generally considered "black boxes" not providing any information about what exactly makes them arrive at their predictions. Since in many applications, e.g. in the medical domain, such lack of transparency may be not acceptable, the development of methods for visualizing, explaining and interpreting deep learning models has recently attracted increasing attention. This introductory paper presents recent developments and applications in this field and makes a plea for a wider use of explainable learning algorithms in
Explainable AI
Wojciech Samek Gregoire Montavon Andrea Vedaldi Lars Kai Hansen Klaus-Robert Muller
152282481
Explainable AI for Trees: From Local Explanations to Global Understanding
Tree based machine learning models such as random forests, decision trees, and gradient boosted trees are the most popular non linear predictive models used in practice today, yet comparatively little attention has been paid to explaining their predictions. Here we significantly improve the interpretability of tree based models through three main contributions: 1) The first polynomial time algorithm to compute optimal explanations based on game theory. 2) A new type of explanation that directly measures local feature interaction effects. 3) A new set of tools for understanding global model structure based on combining many local explanations of each prediction. We apply these tools to three medical machine learning problems and show how combining many high quality local explanations allows us to represent global structure while retaining local faithfulness to the original model. These tools enable us to i) identify high magnitude but low frequency non linear mortality risk factors in the general US population, ii) highlight distinct population sub groups with shared risk characteristics, iii) identify non linear interaction effects among risk factors for chronic kidney disease, and iv) monitor a machine learning model deployed in a hospital by identifying which features are degrading the model's performance over time. Given the popularity of tree based machine learning models, these improvements to their interpretability have implications across a broad set of domains.
ArXiv
Scott M Lundberg Gabriel G Erion Hugh Chen Alex J DeGrave Jordan M Prutkin Bala G Nair Ronit Katz Jonathan Himmelfarb Nisha Bansal Su-In Lee
59606335
Explanation in Human AI Systems: A Literature Meta Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI
This is an integrative review that address the question, "What makes for a good explanation?" with reference to AI systems. Pertinent literatures are vast. Thus, this review is necessarily selective. That said, most of the key concepts and issues are expressed in this Report. The Report encapsulates the history of computer science efforts to create systems that explain and instruct (intelligent tutoring systems and expert systems) The Report expresses the explainability issues and challenges in modern AI, and presents capsule views of the leading psychological theories of explanation. Certain articles stand out by virtue of their particular relevance to XAI, and their methods, results, and key points are highlighted. It is recommended that AI/XAI researchers be encouraged to include in their research reports fuller details on their empirical or experimental methods, in the fashion of experimental psychology research reports: details on Participants, Instructions, Procedures, Tasks, Dependent Variables (operational definitions of the measures and metrics) Independent Variables (conditions) and Control Conditions.
ArXiv
Shane T Mueller Robert R Hoffman William J Clancey Abigail Emrey Gary Klein
58004583
Automated rationale generation: a technique for explainable AI and its effects on human perceptions
Automated rationale generation is an approach for real time explanation generation whereby a computational model learns to translate an autonomous agent's internal state and action data representations into natural language. Training on human explanation data can enable agents to learn to generate human like explanations for their behavior. In this paper, using the context of an agent that plays Frogger, we describe (a) how to collect a corpus of explanations, (b) how to train a neural rationale generator to produce different styles of rationales, and (c) how people perceive these rationales. We conducted two user studies. The first study establishes the plausibility of each type of generated rationale and situates their user perceptions along the dimensions of confidence, humanlike ness, adequate justification, and understandability. The second study further explores user preferences between the generated rationales with regard to confidence in the autonomous agent, communicating failure and unexpected behavior. Overall, we find alignment between the intended differences in features of the generated rationales and the perceived differences by users. Moreover, context permitting, participants preferred detailed rationales to form a stable mental model of the agent's behavior.
IUI
Upol Ehsan Pradyumna Tambwekar Larry Chan Brent Harrison Mark O Riedl
84832474
Can we do better explanations? A proposal of user centered explainable AI
Artificial Intelligence systems are spreading to multiple applications and they are used by a more diverse audience. With this change of the use scenario, AI users will increasingly require explanations. The first part of this paper makes a review of the state of the art of Explainable AI and highlights how the current research is not paying enough attention to whom the explanations are targeted. In the second part of the paper, it is suggested a new explainability pipeline, where users are classified in three main groups (developers or AI researchers, domain experts and lay users) Inspired by the cooperative principles of conversations, it is discussed how creating different explanations for each of the targeted groups can overcome some of the difficulties related to creating good explanations and evaluating them.
IUI Workshops
Mireia Ribera Agata Lapedriza
204832291
On The Role of Knowledge Graphs in Explainable AI
The current hype of Artificial Intelligence (AI) mostly refers to the success of machine learning and its sub domain of deep learning. However, AI is also about other areas, such as Knowledge Representation and Reasoning, or Distributed AI, i.e. areas that need to be combined to reach the level of intelligence initially envisioned in the 1950s. Explainable AI (XAI) now refers to the core backup for industry to apply AI in products at scale, particularly for industries operating with critical systems. This paper reviews XAI not only from a Machine Learning perspective, but also from the other AI research areas, such as AI Planning or Constraint Satisfaction and Search. We expose the XAI challenges of AI fields, their existing approaches, limitations and opportunities for Knowledge Graphs and their underlying technologies.
PROFILES/SEMEX@ISWC
Freddy Lecue
213711544
Explainable ai preface
Wojciech Samek Gregoire Montavon Andrea Vedaldi Lars Kai Hansen Klaus-Robert Muller
195819508
Deep learned faces of pain and emotions: Elucidating the differences of facial expressions with the help of explainable AI methods
Abstract Deep neural networks are successfully used for object and face recognition in images and videos. In order to be able to apply such networks in practice, for example in hospitals as a pain recognition tool, the current procedures are only suitable to a limited extent. The advantage of deep neural methods is that they can learn complex non linear relationships between raw data and target classes without limiting themselves to a set of hand crafted features provided by humans. However, the disadvantage is that due to the complexity of these networks, it is not possible to interpret the knowledge that is stored inside the network. It is a black box learning procedure. Explainable Artificial Intelligence (AI) approaches mitigate this problem by extracting explanations for decisions and representing them in a human interpretable form. The aim of this paper is to investigate the explainable AI methods Layer wise Relevance Propagation (LRP) and Local Interpretable Model agnostic Explanations (LIME) These approaches are applied to explain how a deep neural network distinguishes facial expressions of pain from facial expressions of emotions such as happiness and disgust.
tm Technisches Messen
Katharina Weitz Teena Hassan Ute Schmid Jens-Uwe Garbas
198952499
Explainable AI in Industry
Artificial Intelligence is increasingly playing an integral role in determining our day to day experiences. Moreover, with proliferation of AI based solutions in areas such as hiring, lending, criminal justice, healthcare, and education, the resulting personal and professional implications of AI are far reaching. The dominant role played by AI models in these domains has led to a growing concern regarding potential bias in these models, and a demand for model transparency and interpretability. In addition, model explainability is a prerequisite for building trust and adoption of AI systems in high stakes domains requiring reliability and safety such as healthcare and automated transportation, and critical industrial applications with significant economic implications such as predictive maintenance, exploration of natural resources, and climate change modeling. As a consequence, AI researchers and practitioners have focused their attention on explainable AI to help them better trust and understand models at scale. The challenges for the research community include (i) defining model explainability, (ii) formulating explainability tasks for understanding model behavior and developing solutions for these tasks, and finally (iii) designing measures for evaluating the performance of models in explainability tasks. In this tutorial, we will present an overview of model interpretability and explainability in AI, key regulations/laws, and techniques/tools for providing explainability as part of AI/ML systems. Then, we will focus on the application of explainability techniques in industry, wherein we present practical challenges/ guidelines for using explainability techniques effectively and lessons learned from deploying explainable models for several web scale machine learning and data mining applications. We will present case studies across different companies, spanning application domains such as search and recommendation systems, sales, lending, and fraud detection. Finally, based on our experiences in industry, we will identify open problems and research directions for the data mining/machine learning community.
KDD
Krishna Gade Sahin Cem Geyik Krishnaram Kenthapadi Varun Mithal Ankur Taly