Filtered by themeExplainable AI (XAI)11 papersclear ✕

User Characteristics in Explainable AI: The Rabbit Hole of Personalization?Robert Nimmo · 2024As Artificial Intelligence (AI) becomes ubiquitous, the need for Ex­ plainable AI (XAI) has become critical for transparency and trust among users. A significant challenge in XAI is catering to diverse users, such as data scientists, domain experts, and end-users. Recent research has started to investigate how users’ characteristics impact interactions with and user experience of explanations, with a view to personalizing XAI. However, are we heading down a rabbit hole by focusing on unimportant details? Our research aimed to investi­ gate how user characteristics are related to using, understanding, and trusting an AI system that provides explanations. Our empirical study with 149 participants who interacted with an XAI system that flagged inappropriate comments showed that very few user characteristics mattered; only age and the personality trait open­ ness influenced actual understanding. Our work provides evidence to reorient user-focused XAI research and question the pursuit of personalized XAI based on fine-grained user characteristics. Disclaimer: This paper contains examples of language that some people may find offensive. CCS CONCEPTS • Human-centered computing → Empirical studies in HCI; Interactive systems and tools; • Computing methodologies → Machine learning.

UX of AI & PersonalizationResearch Papers

Article LIME-Mine: Explainable Machine Learning for User Behavior Analysis in IoT ApplicationsXiaobo Cai · 2024In Internet of Things (IoT) applications, user behavior is influenced by factors such as network structure, user activity, and location. Extracting valuable patterns from user activity traces can lead to the development of smarter, more personalized IoT applications and improved user experience. This paper proposes a LIME-based user behavior preference mining algorithm that leverages Explainable AI (XAI) techniques to interpret user behavior data and extract user preferences. By training a black-box neural network model to predict user behavior using LIME and approximating predictions with a local linear model, we identify key features influencing user behavior. This analysis reveals user behavioral patterns and preferences, such as habits at specific times, locations, and device states. Incorporating user behavioral information into the resource scheduling process, combined with a feedback mechanism, establishes an active discovery network of user demand. Our approach, utilizing edge computing capabilities, continuously fine-tunes and optimizes resource scheduling, actively adapting to user perceptions. Experimental results demonstrate the effectiveness of feedback control in satisfying diverse user resource requests, enhancing user satisfaction, and improving system resource utilization.

UX of AI & PersonalizationResearch Papers

A Roadmap of Explainable Artificial Intelligence: Explain to Whom, When, What and How?Ziming Wang · 2024Explainable artificial intelligence (XAI) has gained significant attention, especially in AI-powered autonomous and adaptive systems (AASs). However, a discernible disconnect exists among research efforts across different communities. The machine learning community often overlooks "explaining to whom," while the human-computer interaction community has examined various stakeholders with diverse explanation needs without addressing which XAI methods meet these requirements. Currently, no clear guidance exists on which XAI methods suit which specific stakeholders and their distinct needs. This hinders the achievement of the goal of XAI: providing human users with understandable interpretations. To bridge this gap, this paper presents a comprehensive XAI roadmap. Based on an extensive literature review, the roadmap summarizes different stakeholders, their explanation needs at different stages of the AI system lifecycle, the questions they may pose, and existing XAI methods. Then, by utilizing stakeholders' inquiries as a conduit, the roadmap connects their needs to prevailing XAI methods, providing a guideline to assist researchers and practitioners to determine more easily which XAI methodologies can meet the specific needs of stakeholders in AASs. Finally, the roadmap discusses the limitations of existing XAI methods and outlines directions for future research.

UX of AI & PersonalizationResearch Papers

Predicting the Need for XAI from High-Granularity Interaction DataVagner Figueredo de Santana, Ana Fucs et al. · 2023Recent advances in Artificial Intelligence (AI) and Machine Learning (ML) brought light on the need for explainability in multiple domains (e.g., healthcare, finance, justice, and recruiting). Explainability or Explainable AI (XAI) can be defined as everything that makes AI more understandable to human beings. However, XAI features may vary according to the AI algorithm used. Beyond XAI features, different AI algorithms vary in terms of speed, performance, and costs associated with training/running models. Knowing when to choose the right algorithm for the task at hand, therefore, is fundamental in multiple AI systems, for instance, AutoML and AutoAI. In this paper, we propose a method to analyze patterns of high-granularity user interface (UI) events (i.e., mouse, keyboard, and additional custom events triggered on the millisecond scale) to predict when users will interact with UI elements that provide explainability for the AI in place. In this context, this paper presents: (1) a user study involving 37 participants (7 in the pilot phase and 30 in the main experiment phase) in which people performed a task of reporting a bug using a text form associated with an AI data quality meter and its XAI UI element and (2) an approach to model micro behavior using 𝑛𝑜𝑑𝑒2𝑣𝑒𝑐 to predict when the interaction with XAI UI element will occur. The proposed approach uses a rich dataset (approximately 129k events) and combines 𝑛𝑜𝑑𝑒2𝑣𝑒𝑐and a Logistic Regression classifier. Results obtained show we have obtained an event-by-event prediction of the interaction with XAI with an average F-score of 0.90 (𝜎= 0.06). From the presented results, one expects to support researchers in the realm of UI personalization to consider high-granularity interaction data when predicting the need for XAI while users are interacting with AI model outputs.

UX of AI & PersonalizationResearch Papers

Interpretability, Then What? Editing Machine Learning Models to Reflect Human Knowledge and ValuesAlex Kale · 2022Machine learning (ML) interpretability techniques can reveal unde- sirable patterns in data that models exploit to make predictions— potentially causing harms once deployed. However, how to take action to address these patterns is not always clear. In a collabo- ration between ML and human-computer interaction researchers, physicians, and data scientists, we develop GAM Changer, the first interactive system to help domain experts and data scientists easily and responsibly edit Generalized Additive Models (GAMs) and fix problematic patterns. With novel interaction techniques, our tool puts interpretability into action—empowering users to analyze, val- idate, and align model behaviors with their knowledge and values. Physicians have started to use our tool to investigate and fix pneu- monia and sepsis risk prediction models, and an evaluation with 7 data scientists working in diverse domains highlights that our tool is easy to use, meets their model editing needs, and fits into their current workflows. Built with modern web technologies, our tool runs locally in users’ web browsers or computational notebooks, lowering the barrier to use. GAM Changer is available at the fol- lowing public demo link: https://interpret.ml/gam-changer. CCS CONCEPTS • Computing methodologies →Machine learning.

UX of AI & PersonalizationResearch Papers

Visual, textual or hybrid: the effect of user expertise on different explanationsMaxwell Szymanski · 2021As the use of AI algorithms keeps rising continuously, so does the need for their transparency and accountability. However, literature often adopts a one-size-fits-all approach for developing explana- tions when in practice, the type of explanations needed depends on the type of end-user. This research will look at user expertise as a variable to see how different levels of expertise influence the under- standing of explanations. The first iteration consists of developing two common types of explanations (visual and textual explana- tions) that explain predictions made by a general class of predictive model learners. These explanations are then evaluated by users of different expertise backgrounds to compare the understanding and ease-of-use of each type of explanation with respect to the different expertise groups. Results show strong differences between experts and lay users when using visual and textual explanations, as well as lay users having a preference for visual explanations which they perform significantly worse with. To solve this problem, the second iteration of this research focuses on the shortcomings of the first two explanations and tries to minimize the difference in understanding between both expertise groups. This is done through the means of developing and testing a candidate solution in the form of hybrid explanations, which essentially combine both visual and textual explanations. This hybrid form of explanations shows a significant improvement in terms of correct understanding (for lay users in particular) when compared to visual explanations, whilst not compromising on ease-of-use at the same time. CCS CONCEPTS • Human-centered computing →Empirical studies in visualiza- tion; User studies; User models; • Information systems →De- cision support systems; • Computing methodologies →Arti- ficial intelligence.

UX of AI & PersonalizationResearch Papers

Anchoring Bias Affects Mental Model Formation and User Reliance in Explainable AI SystemsMahsan Nourani · 2021EXplainable Artificial Intelligence (XAI) approaches are used to bring transparency to machine learning and artificial intelligence models, and hence, improve the decision-making process for their end-users. While these methods aim to improve human understand- ing and their mental models, cognitive biases can still influence a user’s mental model and decision-making in ways that system de- signers do not anticipate. This paper presents research on cognitive biases due to ordering effects in intelligent systems. We conducted a controlled user study to understand how the order of observing sys- tem weaknesses and strengths can affect the user’s mental model, task performance, and reliance on the intelligent system, and we investigate the role of explanations in addressing this bias. Using an explainable video activity recognition tool in the cooking domain, we asked participants to verify whether a set of kitchen policies are being followed, with each policy focusing on a weakness or a strength. We controlled the order of the policies and the presence of explanations to test our hypotheses. Our main finding shows that those who observed system strengths early-on were more prone to automation bias and made significantly more errors due to positive first impressions of the system, while they built a more accurate mental model of the system competencies. On the other hand, those who encountered weaknesses earlier made significantly fewer er- rors since they tended to rely more on themselves, while they also underestimated model competencies due to having a more negative first impression of the model. Our work presents strong findings that aim to make intelligent system designers aware of such biases when designing such tools. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. IUI ’21, April 14–17, 2021, College Station, TX, USA © 2021 Association for Computing Machinery. ACM ISBN 978-1-4503-8017-1/21/04...$15.00 https://doi.org/10.1145/3397481.34

UX of AI & PersonalizationResearch Papers

How Do Visual Explanations Foster End Users' Appropriate Trust in Machine Learning?Fumeng Yang · 2020We investigated the effects of example-based explanations for a machine learning classifier on end users’ appropriate trust. We explored the effects of spatial layout and visual representation in an in-person user study with 33 participants. We measured partici- pants’ appropriate trust in the classifier, quantified the effects of different spatial layouts and visual representations, and observed changes in users’ trust over time. The results show that each expla- nation improved users’ trust in the classifier, and the combination of explanation, human, and classification algorithm yielded much better decisions than the human and classification algorithm sepa- rately. Yet these visual explanations lead to different levels of trust and may cause inappropriate trust if an explanation is difficult to un- ∗Fumeng Yang was a PhD intern at Pacific Northwest National Laboratory when conducting this research. †Jean Scholtz retired from Pacific Northwest National Laboratory September 2018. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). IUI ’20, March 17–20, 2020, Cagliari, Italy © 2020 Copyright held by the owner/author(s). ACM ISBN 978-1-4503-7118-6/20/03. [https://doi.org/10.1145/3377325.3377480](https://doi.org/10.1145/3377325.3377480) derstand. Visual representation and performance feedback strongly affect users’ trust, and spatial layout shows a moderate effect. Our results do not support that individual differences (e.g., propensity to trust) affect users’ trust in the classifier. This work advances the state-of-the-art in trust-able machine learning and informs the design and appropriate use of automated systems. CCS CONCEPTS • Human-centered computing →Empirical studies in HCI; Information visualization; Empirical studies in visualization; Visualization design and evaluation methods; • Computing method- ologies →Supervised learning by classification.

UX of AI & PersonalizationResearch Papers

Questioning the AI: Informing Design Practices for Explainable AI User ExperincesLiao · 2020A surge of interest in explainable AI (XAI) has led to a vast collection of algorithmic work on the topic. While many rec- ognize the necessity to incorporate explainability features in AI systems, how to address real-world user needs for under- standing AI remains an open question. By interviewing 20 UX and design practitioners working on various AI products, we seek to identify gaps between the current XAI algorithmic work and practices to create explainable AI products. To do so, we develop an algorithm-informed XAI question bank in which user needs for explainability are represented as proto- typical questions users might ask about the AI, and use it as a study probe. Our work contributes insights into the design space of XAI, informs efforts to support design practices in this space, and identifies opportunities for future XAI work. We also provide an extended XAI question bank and discuss how it can be used for creating user-centered XAI. Author Keywords Explainable AI; human-AI interaction; User experience

UX of AI & PersonalizationResearch Papers

Explanation in artificial intelligence: Insights from the social sciencesTim Miller · 2018Article history: Received 22 June 2017 Received in revised form 17 May 2018 Accepted 16 July 2018 Available online 27 October 2018

UX of AI & PersonalizationResearch Papers

A systematic review and taxonomy of explanations in decision support and recommender systemsIngrid Nunes · 2017With the recent advances in the field of artificial intelligence, an increasing number of decision-making tasks are delegated to software systems. A key requirement for the success and adoption of such systems is that users must trust system choices or even fully automated decisions. To achieve this, explanation facilities have been widely investigated as a means of establishing trust in these systems since the early years of expert systems. With today’s increasingly sophisticated machine learning algorithms, new challenges in the context of explanations, accountability, and trust towards such systems constantly arise. In this work, we systematically review the literature on explanations in advice-giving systems. This is a family of systems that includes recommender systems, which is one of the most successful classes of advice- giving software in practice. We investigate the purposes of explanations as well as how they are generated, presented to users, and evaluated. As a result, we derive a novel comprehensive taxonomy of aspects to be considered when designing explanation facilities for current and future decision support systems. The taxonomy includes a variety of different facets, such as explanation objective, responsiveness, content and presentation. Moreover, we identified several challenges that remain unaddressed so far, for example related to fine-grained issues associated with the presentation of explanations and how explanation facilities are evaluated. B Ingrid Nunes ingridnunes@inf.ufrgs.br Dietmar Jannach dietmar.jannach@tu-dortmund.de 1 Universidade Federal do Rio Grande do Sul (UFRGS), Porto Alegre, Brazil 2 TU Dortmund, Dortmund, Germany 123 394 I. Nunes, D. Jannach

UX of AI & PersonalizationResearch Papers