What Is the Role of Explainability in Medical Artificial Intelligence? A Case-Based Approach

This article reflects on explainability in the context of medical artificial intelligence (AI) applications, focusing on AI-based clinical decision support systems (CDSS). After introducing the concept of explainability in AI and providing a short overview of AI-based clinical decision support systems (CDSSs) and the role of explainability in CDSSs, four use cases of AI-based CDSSs will be presented. The examples were chosen to highlight different types of AI-based CDSSs as well as different types of explanations: a machine language (ML) tool that lacks explainability; an approach with post hoc explanations; a hybrid model that provides medical knowledge-based explanations; and a causal model that involves complex moral concepts. Then, the role, relevance, and implications of explainability in the context of the use cases will be discussed, focusing on seven explainability-related aspects and themes. These are: (1) The addressees of explainability in medical AI; (2) the relevance of explainability for medical decision making; (3) the type of explanation provided; (4) the (often-cited) conflict between explainability and accuracy; (5) epistemic authority and automation bias; (6) Individual preferences and values; (7) patient autonomy and doctor–patient relationships. The case-based discussion reveals that the role and relevance of explainability in AI-based CDSSs varies considerably depending on the tool and use context. While it is plausible to assume that explainability in medical AI has positive implications, empirical data on explainability and explainability-related implications is scarce. Use-case-based studies are needed to investigate not only the technical aspects of explainability but also the perspectives of clinicians and patients on the relevance of explainability and its implications.