Human-AI Collaboration for UX Evaluation: Effects of Explanation and\n Synchronization

Abstract

Analyzing usability test videos is arduous. Although recent research showed\nthe promise of AI in assisting with such tasks, it remains largely unknown how\nAI should be designed to facilitate effective collaboration between user\nexperience (UX) evaluators and AI. Inspired by the concepts of agency and work\ncontext in human and AI collaboration literature, we studied two corresponding\ndesign factors for AI-assisted UX evaluation: explanations and synchronization.\nExplanations allow AI to further inform humans how it identifies UX problems\nfrom a usability test session; synchronization refers to the two ways humans\nand AI collaborate: synchronously and asynchronously. We iteratively designed a\ntool, AI Assistant, with four versions of UIs corresponding to the two levels\nof explanations (with/without) and synchronization (sync/async). By adopting a\nhybrid wizard-of-oz approach to simulating an AI with reasonable performance,\nwe conducted a mixed-method study with 24 UX evaluators identifying UX problems\nfrom usability test videos using AI Assistant. Our quantitative and qualitative\nresults show that AI with explanations, regardless of being presented\nsynchronously or asynchronously, provided better support for UX evaluators'\nanalysis and was perceived more positively; when without explanations,\nsynchronous AI better improved UX evaluators' performance and engagement\ncompared to the asynchronous AI. Lastly, we present the design implications for\nAI-assisted UX evaluation and facilitating more effective human-AI\ncollaboration.\n

Ähnliche Arbeiten