How Do Developers Use ChatGPT for Software Test Generation? Usage Patterns, Satisfaction, and Code Adoption Analysis

Abstract

Large Language Models (LLMs) such as ChatGPT are increasingly used by developers to generate test code, yet little is known about their real-world usage patterns, developer satisfaction, and the extent to which generated code is integrated into software projects. This paper presents an empirical study of 407 user-ChatGPT conversations related to test generation from GitHub, including 152 that contain both test and corresponding source code. We systematically categorize the primary use cases of ChatGPT in test generation, employ sentiment analysis to assess developer satisfaction across different testing scenarios, and analyze the adoption between ChatGPT-generated test code and existing project code using PMD’s Copy-Paste Detector (CPD) tool. Our analysis identifies six ChatGPT usage scenarios in test generation: test case creation, framework demonstration, test enhancement, debugging, CI/CD integration, and mocking solution. Among these, test case creation and framework demonstration emerge as the most frequent. Sentiment analysis reveals that, except for framework demonstration, all other scenarios exhibit a higher proportion of negative feedback than positive. Moreover, adoption analysis indicates that only 25.7% of conversations show evidence of code reuse, suggesting developers are generally cautious when integrating LLM-generated test code. Based on these findings, we propose several practical implications for enhancing the effectiveness of LLM-assisted test generation.

Ähnliche Arbeiten