Who’s the Author? How Explanations Impact User Reliance in AI-Assisted Authorship Attribution

Abstract

Despite growing interest in explainable NLP, it remains unclear how explanation strategies shape user behavior in tasks like authorship identification, where relevant textual features may be difficult for lay users to pinpoint.To support their analysis of text style, we consider two explanation types: example-based style rewrites and feature-based rationales, generated using a LLM-based pipeline.We measured how explanations impact user behavior in a controlled study (n=95) where participants completed authorship identification tasks with our types of assistance.While no explanation type improved overall task accuracy, fine-grained reliance patterns (Schemmer et al., 2023) revealed that rewrites supported appropriate reliance, whereas presenting both explanation types increased AI overreliance, minimizing participant self-reliance.We find that participants exhibiting better reliance behaviors had focused explanation needs, contrasting with the diffused preferences of those who overrelied on AI, or incorrectly self-relied.These findings highlight the need for adaptive explanation systems that tailor support based on specific user reliance behaviors.

Ähnliche Arbeiten